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The  purpose  of  this  thesis  was  to  identify  three 
promising  least  squares  selection  procedures  discussed  in 
the  literature  during  the  previous  decade  and  then  test  them 
using  simulation.  The  three  criteria  chosen  for  this  study 
were  minimum  mean  square  error  (Min  MSE),  minimum  Sp,  and 
minimum  C^. 

Most  of  the  previous  simulations  in  this  area  are  limited 
to  investigating  the  usefulness  of  variable  selection 
criteria  when  all  relevant  regressors  and  some  noise 
variables  are  available.  It  is  questionable  whether  all 
relevant  variables  will  be  included.  This  research  has 
examined  the  effects  of  not  including  a  significant  variable 
in  the  variable  pool. 

In  examining  each  criterion,  emphasis  was  placed  on  the 
technique's  performance  under  varying  amounts  of 
mu 1 t icol 1 inea r i ty ,  variable  variation,  number  of  variables, 
and  sample  size.  Response  Surface  Methodology  was  used  to 
determine  the  effects  of  varying  these  factors.  A 
comparison  was  then  made  using  the  results  from  the  Response 
Surface . 

To  supplement  the  simulation  research  a  comprehensive 
literature  review  of  the  most  current  journal  articles 


dealing  with  several  least  squares  criteria  has  been 
provided-  This  review  includes  a  discussion  of  each 
technique's  strengths  and  weaknesses.  Since  many  of  the 
least  squares  variable  selection  criteria  are  addressed, 
this  thesis  serves  as  a  useful  starting  place  for  various 
regression  questions. 
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A  COMPARISON  OF  VARIABLE  SELECTION  CRITERIA 


FOR  MULTIPLE  LINEAR  REGRESSION:  A  SIMULATION  STUDY 


I . 


Introduct ion 


Background 

Linear  regression  is  a  statistical  tool  used  to  fit  data 
to  a  surface.  From  the  surface,  predicted  values  for  the 
dependent  variable  are  determined  with  confidence. 
Unfortunately,  a  difficult  aspect  of  regression  analysis  is 
determining  the  best  set  of  independent  variables  to  include 
in  the  linear  regression  model. 

Before  the  wide  use  of  computers,  the  analyst  was  forced 
to  rely  on  his  intuition  for  determining  which  variables 
should  be  included  in  the  model.  Variable  selection 
techniques  were  available  but  difficult  to  implement.  It 
was  important  for  the  analyst  to  "screen"  the  variables 
first  by  a  reasonability  test.  For  example,  say  the 
dependent  variable  is  the  height  of  a  man.  Possible 
independent  variables  are:  size  of  hand,  size  of  feet,  and 
blood-type.  It  seems  reasonable  to  assume  that  the  height 


of  a  man  is  dependent  on  the  size  of  his  hands  and  feet. 
But,  it  does  not  make  sense  to  assume  his  blood-type 
significantly  contributes  to  his  height.  Therefore,  the 
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analyst  would  include  only  feet  and  hand  size  into  his 
variable  pool.  The  analyst  could  perform  his  analysis  on 
all  the  variables  in  the  variable  pool,  which  is  ordinary 
least  squares  (OLS),  or  he  could  use  a  variable  selection 
method  to  find  the  most  significant  variables.  If  he  simply 
calculates  the  regression  equation  containing  both  feet  size 
and  hand  size,  the  method  is  called  OLS.  A  better  model 
might  include  only  one  of  these  variables,  so  the  analyst 
might  want  to  use  one  of  the  selection  criteria  on  the 
variable  pool . 

Due  to  computer  limitations,  only  a  few  criteria  for 
selecting  variables  were  available,  and  the  literature  out¬ 
paced  implementation.  With  computer  advancements, 
implementation  of  new  techniques  should  be  increasing; 
however,  this  is  not  necessarily  true.  Often,  these 
published  techniques  are  forgotten.  Rarely  are  the  "new 
techniques"  tested  and  implemented.  In  fact,  most  of  the 
techniques  currently  in  use  are  1970  vintage,  or  earlier. 

To  be  sure,  some  of  these  "new  techniques"  are  superior  to 
those  in  use.  Since  these  criteria  are  lost  in  the 
literature,  research  may  be  suffering. 

Several  problems  can  occur  if  the  variable  selection  is 
left  solely  to  the  computer.  First,  there  are  literally 
hundreds  of  variable  selection  criteria  available.  Under 
certain  circumstances,  one  method  may  be  superior  to 
another.  By  mindlessly  using  a  technique,  the  resulting 
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model  might  be  worthless.  Certainly  there  are  methods  which 
"screen"  better  than  others.  However,  there  is  not  a  method 
which  absolutely  picks  only  true  variables.  During  a 
simulation  study.  Flack  and  Chang  demonstrated  that  when  the 
variable  pool  contains  true  variables,  those  which 
significantly  contribute  to  the  dependent  variable,  and 
random  noise  variables,  random  noise  variables  were  often 
chosen  (8:85).  Only  under  the  most  ideal  circumstance  were 
a  fair  amount  of  true  variables  chosen. 

The  problem  of  selecting  noise  terms  was  further 
illustrated  by  Freedman.  (9)  By  constructing  simulated  data 
containing  only  noise  variables,  Freedman  demonstrated  that 
high  values  could  result  when  a  model  contained  variables 
which  are  theoretically  independent  of  the  dependent 
variable.  (9:153)  The  point  being,  selection  criteria  are 
not  fail  proof.  The  model  chosen  could  possibly  contain 
several  noise  variables.  The  more  noise  variables  in  the 
variable  pool  (those  not  screened  by  other  means),  the  more 
likely  one  or  more  will  be  included  in  the  model. 

A  subsequent  simulation  study  performed  by  Hoerl,  Hoerl, 
and  Schuenemeyer  identified  and  tested  the  effects  of  sample 
size,  random  noise  variables  and  correlated  data  on  biased 
regression  techniques  and  least  squares  regression.  (13) 
Their  results  indicated  that  both  biased  and  least  squares 
regression  performed  equally  well  except  for  stepwise 


regression  and  principle  component  regression  which 
performed  poorly.  (13:369) 

Objective 

The  objective  of  this  research  is  to  identify  three 
promising  least  squares  selection  procedures  discussed  in 
the  literature  during  the  previous  decade  and  then  test  them 
using  simulation.  The  simulation  wa s  similar  to  Flack  and 
Chang's  in  the  sense  that  the  effects  of  noise  variables  and 
sample  size  were  examined  (8:84-86),  but  did  not  make  the 
same  assumptions  and  used  different  criteria.  Flack  and 
Chang  tested  the  R  and  stepwise  procedures.  Like  several 
other  simulations,  they  included  all  the  true  variables  and 
some  noise  variables  and  then  determined  how  many  noise 
variables  were  selected.  This  research  examined  three 
criteria,  minimum  mean  square  error  (Min  MSE) ,  minimum  Sp, 
and  minimum  Cp  and  included  random  noise  variables.  An 
extension  of  Flack  and  Chang's  work  has  been  made.  Flack 
and  Chang's  research  did  not  examine  the  problem  of  not 
including  all  the  significant  variables.  In  the  previous 
example,  data  on  the  height,  size  of  feet,  size  of  hands, 
and  blood-type  of  a  man  was  collected.  It  is  possible  that 
the  weight  of  a  man  is  a  also  a  significant  factor  in 
determining  the  man's  height,  but  that  no  information  has 
been  collected  on  the  man's  weight.  Most  criteria  are  based 
on  the  assumption  that  all  relevant  variables  are  included 
in  the  variable  pool;  however,  the  fact  is,  all  relevant 
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variables  may  not  be  included.  This  research  has  examined 
the  effects  of  not  including  a  significant  variable  in  the 
variable  pool. 

In  examining  each  criterion,  emphasis  was  placed  on  the 
technique's  performance  under  varying  amounts  of 
mu  1 1 i col 1 ine ra r i ty ,  variable  variation,  number  of  variables, 
and  sample  size.  Response  Surface  Methodology  was  used  to 
determine  the  effects  of  varying  these  factors.  A 
comparison  was  then  made  using  the  results  from  the  Response 
Surface . 

Previous  simulations  have  based  their  comparisons  solely 
on  the  number  of  noise  variables  chosen.  ( 2 ; 8 ; 1 2 )  The 
weakness  of  this  performance  measure  lies  in  the  fact  that 
as  the  number  of  variables  in  the  variable  pool  changes,  the 
likelihood  of  choosing  solely  the  correct  variables  also 
changes.  In  this  study  an  alternative  performance  measure 
was  be  used  for  comparison. 

To  supplement  the  simulation  research  a  comprehensive 
literature  review  of  the  most  current  journal  articles 
dealing  with  several  least  squares  criteria  is  provided. 

This  review  includes  a  discussion  of  each  technique's 
strengths  and  weaknesses.  Since  many  of  the  least  squares 
variable  selection  criteria  are  addressed,  this  thesis 
serves  as  a  useful  starting  point  for  various  regression 
questions . 


II.  Concept  Overview 


Least  Squares  Regression 

Assumptions .  Assumptions  are  made  prior  to  constructing 
a  least  squares  linear  regression.  First,  assume  the 
collected  data  represents  the  population  it  came  from.  That 
is,  the  data  reflects  the  normal  case  of  the  variable. 
Second,  assume  the  error  terms  are  independent  and 
identically  distributed,  from  a  normal  distribution,  with  a 
mean  of  zero  and  variance  o  . 

Notation .  The  goal  in  linear  regression  is  to  find  the 
"best-subset"  of  independent  variables  to  include  in  the 
model  which  adequately  predicts  the  value  of  the  dependent 
variable.  In  general,  the  linear  least  squares  regression 
equation  is  written  in  the  following  manner: 

Y  =  +  BjX,  +  BjX^  +...+  B^X^  +  G  (1) 

where  Y  is  the  observed  value  of  the  independent 
variable. 

Bq  is  the  constant  term. 

B,,B2,...,B^  are  the  constant  terms  for  the 
dependent  variables  X, ,  X^ ,  .  •  •  ,  X,^. 

k  is  the  number  of  independent  variables  included 
in  the  model. 

G  is  the  error  term. 

If  there  are  n  observations,  or  data  points,  the  above 
equation  may  be  written  as: 
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(2) 


E  Y,  =  Z  (Be,  +  +  ...  +  B,^X,,  +  e,  ) 

For  convenience,  the  above  equation  can  be  written  in  matrix 
notation . 

Y  =  X  B  +  €  (  3  ) 

where  Y  is  a  (n  x  1 )  column  vector: 


Y  = 


Y, 

y' 


and  X  is  a  (n  x  (k+1))  matrix. 


1 

X,, 

X-,2 

« 

• 

•  Xi,^ 

1 

Xj, 

Xj, 

• 

• 

.  Xj,. 

1  X,,  X,2 


The  first  column  contains  all  ones  for  the  constant  terms. 
The  remaining  columns  contain  the  X^^  independent  variables. 
The  X  matrix  is  commonly  referred  to  as  the  design  matrix. 

B  is  a  (k  X  1 )  column  vector: 
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and  E  is  a  (n  x  1)  column  vector: 


€  = 


In  least  squares  regression,  each  subset  of  regression 
variables  generates  a  surface  which  minimizes  the  squared 
distance  (error)  between  the  observed  values  for  the 
dependent  variables,  Y,  and  the  predicted  values  for  the 
dependent  variable 

min  E  =  min  S  (^,  -  Y  )  ^  ( 4 ) 

The  goal  is  to  find  the  subset  of  variables  which  minimizes 
the  squared  distances  between  the  actual  values  observed  and 
the  fitted  surface.  The  sum  of  the  squared-error  values  is 
commonly  referred  to  as  the  sum-of -squares  error  (SSE). 

Graphically,  in  two  dimensions  a  regression  resembles  the 
following : 
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Figure  1.  Two-dimensional  representation 
of  linear  least  squares  regression 


where 


SSR  is  the  sum  of  the  squared  distances  from  the  mean 
to  the  regression  line  and  is  called  Regression 
Sum  of  Squares. 

SSE  is  the  sum  of  the  squared  distances  from  the  point 
to  the  regression  line  and  is  called  Sum  of 
Squares  Error. 

SST  is  the  sum  of  SSR  and  SSE  and  called  Sum  of 
Squares  Total. 


SST 


SSR  +  SSE 


(5) 
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III.  Review  of  Pa st  Research 


Scope 

The  main  focus  of  this  literature  search  is  limited  to 
variable  selection  techniques  published  since  1978.  This 
limitation  was  chosen  for  three  reasons.  First,  two  GOR 
students,  Captain  Larry  J.  Pulcher  (24:1-137),  and  Second 
Lieutenant  Joseph  R.  Cafarella,  Jr.  (4:1-127),  performed 
literature  reviews  on  variable  selection  techniques  for 
their  theses,  written  in  1978  and  1979  respectively. 

Second,  most  of  the  implemented  techniques  were  published 
prior  to  1978.  Third,  an  excellent  review  of  the  pre-1980 
techniques  can  be  found  in  Hocking’ s  1976  paper  (12). 

Method  of  Treatment  and  Organization 

This  literature  review  reports  methods  for  selecting  the 
number  of  variables  to  include  in  a  linear  regression 
equation,  as  well  as  specific  techniques  for  selecting  the 
"best-subset”  of  variables  to  include  in  the  regression 
equat i on . 

Review 

If  the  true  variance,  o^,  of  the  independent  variables  is 
known,  selecting  the  proper  regression  model  is  relatively 
simple  (7:294).  However,  the  true  variance  is  rarely  known 
a  priori.  It  is  more  likely  that  the  underlying  variance 
will  have  to  be  estimated.  Therefore,  techniques  must  be 
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available  for  the  analyst  so  that  he  may  properly  select  the 
variables  to  include  in  a  linear  regression  equation. 

Number  of  Variables.  Draper  and  Smith  (7:294)  discuss 
two  opposing  viewpoints  concerning  the  proper  number  of 
independent  variables  (or  regressors)  to  include  in  a  linear 
regression  equation.  One  viewpoint  is  all  variables  should 
be  included  for  both  a  reliable  fit  and  predictive  purposes. 
The  opposite  point-of -view  is  to  include  as  few  variables  as 
possible,  which  adequately  predict,  in  the  model.  The 
latter  point-of -view  is  commonly  referred  to  as  parsimony. 

For  predictive  purposes,  Bayesian  statisticians  recommend 
including  all  possible  variables  (28:1553).  They  argue  the 
cost  associated  with  collecting  and  maintaining  the  data 
must  be  considered.  If  these  costs  are  zero,  than  every 
variable  should  be  included.  Only  by  including  as  many 
variables  as  possible  can  one  be  certain  to  obtain  the  best 
prediction . 

Cafarella  (4:14)  points  out  the  major  flaw  of  including 
every  possible  variable.  The  resulting  model  over-fits  the 
data.  That  is,  the  independent  variables  not  only  predict 
the  dependent  variable,  but  also  predict  the  variation,  or 
noise.  The  resulting  equation  is  useful  for  predicting  the 
past,  but  not  very  useful  for  predicting  the  future.  Thus, 
it  is  arguable  whether  including  all  possible  regressors  is 


the  best  course  of  action. 


Even  the  Bayesian  statisticians  agree  that  i£  the  costs 
of  collecting  and  maintaining  data  are  high,  then  one  should 
include  as  few  variables  as  possible  in  the  model.  However, 
some  believe  in  the  concept  of  parsimony  regardless  of  the 
costs.  Parsimony  is  predicting  as  much  as  possible,  with 
as  little  as  possible.  Trader  states,  "Variable  selection 
techniques,  which  facilitate  parsimony,  have  been  viewed 
primarily  as  providing  a pproxima te  representations  of  a  true 
underlying  process."  (28:1553)  The  advantages  of  parsimony 
is  that  model  does  not  over-fit  the  data,  and  it  does  not 
cost  as  much  as  a  larger  model  to  collect  and  maintain. 
However,  the  question  remains:  does  the  model  contain  enough 
variables  to  predict  adequately? 

Recent  literature  (23:509-16;  3:131-6;  24:45-54)  suggests 

a  method  for  determining  the  optimal  number  of  regressors 
for  a  linear  regression  equation.  The  two  driving  factors 
for  determining  the  optimal  number  of  regressors  are  sample 
size  and  the  mean  square  error  of  prediction  (MSEP). 

(23:514)  MSEP  is  the  squared  difference  between  the 
predicted  value,  ,  and  the  actual  value,  ,  divided  by 

the  degrees  of  freedom.  The  MSEP  has  been  decomposed  by 
Breiman  and  Freedman  (3:131)  into  the  conditional  MSEP,  and 
the  unconditional  MSEP.  The  conditional  MSEP  is 

M  .  M^p  =  /  Y,  and  for  all  j 

and  i  *  1 , . . . , n } 


(6) 


while  the  unconditional  MSEP  is  defined  as 

U  =  =  E  { M.,„  ]  (  7  ) 

With  the  mean  square  error  of  prediction  and  a  given 

sample  size,  one  can  find  the  optimal  number  of  parameters, 

« 

p  ,  to  include  in  the  model. 

p  =  k  +  1  ( ® ) 

where 

p  is  the  number  of  parameters  in  the  model 
with  Min  MSEP. 

k  is  the  number  of  independent  variables. 

When  either  M  or  U  from  Eq  (5)  and  Eq  (6)  are  minimized, 
the  corresponding  p  is  the  optimal  number  of  parameters  for 
the  regression  model.  It  is  apparent  that  when  the  sample 
size  is  small,  the  optimal  number  of  parameters  tends  to  be 
small  (28:1562). 

Choosing  the  Best  Subset  of  Variables.  Regardless  of 
whether  the  optimal  number  of  parameters  has  been 
determined,  a  criterion  must  be  used  to  select  which 
variables  should  be  included  in  the  regression  equation. 
Miller  separates  possible  criteria  into  "(i)  those  which 
guarantee  to  find  the  best  fitting  subset  of  some  of  all 
sizes,  and  (ii)  the  'cheap'  methods  which  sometimes  find  the 
best-fitting  subsets."  (20:391) 

"Guaranteed  Methods".  Miller  specifically  mentions 
two  methods  which  guarantee  choosing  the  best-fitting 


variables.  These  two  methods  are  all-subset  and  branch-and- 
bound . 

All-Subsets .  In  the  1984  article,  "Selection 
of  Subsets  of  Regression  Variables"  (20:391),  Miller  claims 
that  only  by  an  exhaustive  search  of  all  2  -1  possible 
subsets  can  one  be  assured  of  finding  the  best-fitting 
model.  He  further  suggests  the  all-subset  procedure  is 
practical  only  when  the  number  of  possible  variables  is  less 
than  twenty.  With  the  improvement  of  computers  over  the 
last  five  years,  it  might  be  feasible  to  test  every  subset 
for  more  than  twenty  variables.  However,  the  necessary 
number  of  regressions  increases  rapidly.  For  instance,  with 
twenty  variables,  over  one  million  subsets  must  be  fit.  By 
increasing  the  number  of  variables  by  just  five  to  twenty- 
five,  the  number  of  necessary  regressions  increases  to 
roughly  33.5  million. 

The  all-subset  procedure  is  not  without  criticism.  One 
of  the  underlying  assumptions  in  regression  is  the  variable 
pool  contains  all  the  relevant  variables  and  some  extraneous 
variables.  (21:160)  However,  it  is  possible  some  of  the 
relevant  variables  are  not  included  in  the  variable  pool. 
Because  of  this,  the  all-subset  procedure  may  not  find  the 
best  model  in  population  terms,  but  always  finds  the  best 
model  within  the  variable  pool  (2:3).  In  fact,  bac)tward 
elimination,  (one  of  the  "cheap  methods,"  which  will  be 
discussed  later),  does  slightly  better  than  all-subsets 


under  certain  circumstances.  Although  Berk  criticizes  the 
all-subset  procedure  under  certain  circumstances,  he  notes 
that  it  usually  does  well  (2:3).  With  large  sample  sizes 
(approaching  the  population  size)  all-subset  is  the  best. 

Branch-and-Bound.  Miller  includes  the  branch- 
and-bound  technique  as  one  of  the  guaranteed  methods  when 
the  optimal  number  of  variables  has  been  determined 
(20:391).  The  branch-and-bound  technique  computes  only  a 
fraction  of  the  possible  models.  It  eliminates  subsets  from 
consideration  if  a  calculated  subset  fits  better  than  the 
eliminated  subset  could  possibly  obtain.  The  eliminated 
subsets  are  said  to  be  "dominated."  With  some  of  the  models 
"dominated,"  the  all-possible  subset  procedure  can  be 
implemented  on  the  subsets  that  are  not  eliminated  by 
branch-and-bound.  The  branch-and-bound  technique  is  not  only 
useful  in  eliminating  possible  subsets,  but  can  be  modified 
to  be  an  efficient  all-subset  procedure  (10:510).  The 
necessary  modification  requires  that  the  "testing"  mechanism 
be  turned  off. 

"Cheap-Methods."  Miller  conveniently  labels  all 
techniques  other  than  the  all-subset  and  the  branch-and- 
bound  procedures  as  "cheap."  These  techniques  are  cheap  in 
the  sense  that  they  do  not  require  as  many  calculations  for 
finding  a  solution.  Consequently,  none  of  the  cheap-methods 
are  guaranteed  to  find  the  best-fitting  model. 


Backward  Elimination . 


Backward  elimination  is 


a  technique  which  considers  all  of  the  possible  variables  in 
the  original  model,  then  eliminates,  one  at  a  time,  those 
variables  which  do  not  significantly  contribute  to  the 
regression.  According  to  Draper  and  Smith,  "this  [backward 
elimination]  is  a  satisfactory  procedure,  especially  for 
statisticians  who  like  to  see  all  the  variables  in  the 
equation  once  in  order  'not  to  miss  anything'."  (7:307) 

This  procedure  is  satisfactory  if  the  data  is  not  highly 
correlated. 

Nonetheless,  backward  elimination  has  its  drawbacks.  One 
major  drawback  is  once  a  variable  is  eliminated,  it  is  not 
considered  again.  It  is  possible  that  an  eliminated 
variable,  which  did  not  look  significant  when  it  was 
eliminated,  will  look  significant  after  other  variables  are 
eliminated.  Another  criticism  of  backward  elimination  is 
that  it  is  not  useable  when  the  number  of  possible  variables 
exceeds  the  number  of  data  points.  Due  to  the  matrix 
manipulations  necessary  in  regression,  this  procedure  cannot 
be  used  when  variables  exceed  data  points. 

Forward  Selection.  Forward  Selection  is 
similar  to  backward  elimination.  However,  instead  of  all  of 
the  variables  being  included  in  the  original  model,  then 
eliminating  one  variable  at  a  time,  forward  selection  starts 
with  one  variable,  and  includes  a  variable-at-a-time.  The 
major  advantage  of  this  technique  is  that  it  can  be 


implemented  when  tl  e  number  of  possible  variables  exceeds 
the  sample  size.  This  technique  can  be  used  since  the 
original  model  has  one  variable,  to  which  variables  are 
subsequently  added.  (20:392). 

The  disadvantages  of  forward  selection  are  similar  to 
backward  elimination.  Like  backward  elimination,  forward 
selection  performs  poorly  when  variables  are  highly 
correlated.  Also,  once  a  variable  enters  the  regression 
equation,  it  is  no  longer  considered  for  elimination. 

Stepwise  Regression.  Stepwise  regression  is  a 
procedure  which  combines  both  the  forward  and  backward 
selection  procedures.  Like  forward  selection,  variables  are 
selected  to  enter  one  at  a  time.  However,  after  a  variable 
enters  the  regression  equation,  all  variables  in  the  subset 
are  reconsidered  for  elimination.  This  improvement  allows 
for  the  deletion  of  those  variables  which  do  not 
significantly  contribute  to  the  model. 

Draper  and  Smith  recommend  this  technique.  However,  they 
add  the  caveat  that  it  is  not  be  used  blindly  (7:310).  If 
stepwise  regression  is  used  with  little  thought,  many  random 
noise  variables  can  be  selected.  As  is  the  case  with  all  of 
the  techniques,  the  likelihood  of  selecting  a  noise  variable 
is  fairly  high  if  the  number  of  noise  variables  is  high. 

To  combat  this  problem  Miller  suggests  an  alternative 
stopping  criterion  for  stepwise  regression.  His  suggestion 
is  to  add  "known"  extraneous  variables  to  the  variable  pool 


» 


(randomly  generated  noise)  then  execute  the  stepwise 
regression  procedure.  Once  one  of  the  "known"  extraneous 
variables  is  selected,  terminate  the  selection  process. 
(20:395)  The  concept  behind  his  suggestion  is  that  once  a 
"known"  extraneous  variable  (one  that  was  augmented  to  the 
variable  pool)  is  selected,  there  is  no  more  useful 
information  to  be  gained  by  adding  another  variable.  The 
model  selected  will  include  all  variables  picked  not 
including  the  known  extraneous  variable.  For  the  technique 
to  be  useful,  Miller  recommends  that  the  number  of  added 
extraneous  variables  should  be  approximately  equal  to  number 
of  variables  already  in  the  variable  pool.  (20:395) 

Excluding  its  weakness  when  several  extraneous  variables 
are  in  the  variable  pool,  stepwise  regression  performs  well 
when  there  is  little  correlation.  Many  computer  packages 
use  a  significance  level  of  0.05  for  forward,  backward,  and 
stepwise  procedure.  The  choice  of  0.05  seems  to  be 
arbitrary  and  unjustified.  In  fact,  the  results  of  a 
simulation  study  conducted  by  Hoerl,  Hoerl,  and  Schuenemeyer 
indicate  that  a  significance  level  of  0.05  often  leads  to 
poor  results  (13:378).  They  demonstrated  that  many  of  the 
true  variables  are  not  selected  when  the  significance  level 
is  0.05.  However,  most  variables  are  picked  using  a 
significance  level  between  0.15  and  0.25 

Near- Optimal-Model.  Narula  and  Wellington 


(21:169)  propose  the  near-best-model  for  Mean  Square 


Absolute  Errors  (MSAE)  in  1983.  MSAE  is  the  sum  of  the 


n  is  the  sample  size. 

p  is  the  number  of  parameters  in  the  model. 

Y  is  the  predicted  value  for  Y. 

Y  is  the  actual  value  of  Y. 

Narula  and  Wellington  put  a  "lower  and  an  upper  limit  on  the 
number  of  variables  to  be  included  in  an  effort  to 


accelerate  the  implicit  enumeration  algorithms  and  aid  the 
investigator  in  selecting  a  model."  (21:169)  Although  they 
apply  the  near-optimal  technique  to  MSAE,  they  suggest  it 
may  be  effective  in  the  least  squares  case. 

Similar  to  Narula  and  Wellington's  proposed  near-best- 
model  for  Mean  Square  Absolute  Errors,  Huang  and 
PanchapaResan  have  implemented  a  procedure  to  eliminate 
inferior  models  with  some  guaranteed  probability  (14:753). 
Their  procedure  is  based  on  the  residual  sums  of  squares  and 
is  similar  to  Furnival  and  Wilson’s  branch-and-bound 


technique.  The  near-best-model  for  Mean  Square  Absolute 
Errors  is  based  on  SSR  making  it  easy  to  implement.  What 
makes  this  technique  especially  appealing  is  that  once  a 


model  is  rejected,  then  all  combinations  of  its  variables 
are  also  eliminated.  (14:758). 

An  example  of  how  the  near-best-method  worlds  is  the 
following : 

Suppose  the  subset  of  variables  under  consideration  is 
{X,,  X~,  Xj)  .  If  this  subset  of  variables  is  rejected,  then 
all  subsets  of  X^ ,  X^,  and  Xj  would  also  be  rejected  (e.g. 

{X,  },  {Xj},  {Xj  },  {X  ,X2  },  etc.). 

Mallows  C-.  Mallows  C.  is  a  statistic  used  to 

“  P  P 

determine  the  best  model  when  the  independent  variables  are 
fixed.  is  an  approximation  of  MSEP. 

SSR 

C.  =  -  +  2p  -  n  (10) 

s' 

where  SSR  is  the  Regression  Sum  of  Squares 
s'  is  the  estimate  for  the  variance 
p  is  the  number  of  parameters 

n  is  the  number  of  data  points 

Theoretically  the  value  of  is  p.  Therefore,  when  Cp  is 
approximately  equal  to  p,  the  model  is  good.  Draper  and 
Smith  suggest  using  this  criterion  in  conjunction  with 
stepwise  regression  to  obtain  the  best  subset  (7:341).  It 
should  be  noted,  however,  as  the  variance  approaches  zero, 
the  Cp  statistic  can  not  be  calculated.  Therefore  this 
method  has  limitations  especially  when  the  fit  is  perfect. 
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Barr  pointed  out  a  weakness  of  Mallows  Cp.  Since  s^,  used 
in  the  Cp  statistic,  is  estimated  from  the  original  variable 
pool,  it  could  be  biased  and  larger  than  the  true  variance. 
(1:5)  If  this  is  case,  the  statistic  will  be  deflated 

S' 

causing  the  wrong  model  to  be  selected. 

A  limitation  of  C^,  as  well  as  many  other  statistics,  is 
that  it  "dependCs]  on  the  observed  data  only  through 
sufficient  statistics,  so  they  model  average  behavior  of  the 
fit  of  a  model  to  the  data."  (30:27)  Weisberg  developed  a 
procedure  which  allocates  the  statistic  to  individual 
cases.  The  advantage  of  Weisberg' s  procedure  is  if  the 
model  under  consideration  is  biased,  it  provides  a  means  to 
determine  the  bias  of  using  a  subset  model  instead  of  the 
entire  model  (30:28) 

Another  application  of  the  Cp  statistic  is  to  choose  the 
model  which  has  the  smallest  C^  value,  (15:863)  By 
choosing  the  model  with  the  minimum  Cp,  it  is  believed  that 
one  is  choosing  the  model  with  the  minimum  prediction  error. 
This  is  appealing,  especially  when  it  is  difficult  to 
determine  the  optimal  subset  using  the  Cp  close  to  p 
criterion.  Since  the  Min  C^  criterion  is  based  on  minimum 
prediction  error  it  is  based  on  a  sound  principle.  However, 
like  the  Cp  close  to  p  criterion,  Min  Cp  is  derived  under 
the  assumption  that  the  independent  variables  are  fixed. 
Since  this  rarely  happens  in  practice,  there  is  some 
question  to  the  usefulness  of  the  Min  Cp  criterion.  Judge, 


Griffiths,  Carter,  Lutkepohl,  and  Lee  recommend  that  the  Min 


Cp  procedure  should  not  be  used  in  any  applied  work. 
(15:864) 


Coefficient  of  Determination.  The  coefficient 
of  determination,  R  ,  is  a  statistic  which  gives  an 
estimation  of  the  amount  of  variation  about  the  mean  which 
is  explained  by  the  model. 

E  (Y.  -  Y)^ 

r2  =  - : -  (11) 


E  (Y,  -  Y)^ 


where 

A 

Y,  is  the  predicted  value  of  Y,. 

Y,  is  the  actual  value  of  Yj. 

Y  is  the  mean  of  Y. 

At  first  one  might  believe  that  it  is  desirable  to  find  the 
model  which  has  the  maximum  R^,  since  it  explains  the  most 

variation  about  the  mean.  However,  this  is  not  necessarily 

2 

the  best  true.  Certainly  when  we  look  at  the  R  value  we 
would  like  to  see  a  large  value,  but  it  should  not  be  used 
as  the  only  measure  for  subset  selection.  Maximum  R^ 
receives  little  praise  as  far  as  its  usefulness  in 
determining  a  good  fit.  The  major  pitfall  of  using  R^  is 
that  whenever  a  variable  is  added,  it  will  increase  R^.  R^ 
will  increase  regardless  of  whether  the  variable  has 
anything  to  do  with  the  dependent  variable.  According  to 
Healy  1986,  "In  particular,  the  multiple  correlation 
coefficient  is  not  really  a  regression-related  concept  at 


all*  It  is  basically  defined  to  be  the  largest  possible 
correlation  between  the  y-variate  and  any  linear  function  of 
the  x's  and  this  only  makes  sense  when  y  and  x's  have  a 
joint  probability  distribution.”  (11:1984)  If  maximum  is 
used  as  the  selection  criterion,  the  model  containing  all 

variables  will  always  be  selected. 

2 

Maximum  Adiusted  R  or  Minimum  MSB.  For 
simplicity  only  maximum  Adjusted  R^  will  be  discussed. 
However,  maximum  Adjusted  R^  and  Minimum  MSE  test  exactly 
the  same  thing. 

Adjusted  R^  is  related  to  R^,  but  an  adjustment  has  been 
made  for  the  degrees  of  freedom.  The  following  equation 
shows  the  relationship  between  R^  and  Adjusted  R^. 

(1-R^)(n-1) 

Adjusted  R‘^  =  1  -  -  (12) 

(  n  -  p ) 

According  to  Draper  and  Smith,  the  adjusted  R^  statistic 
can  be  used  not  only  to  compare  models  for  the  same  data  set 
(the  same  variable  selection  discussed  in  all  other  sections 
of  this  literature  review) ,  but  also  to  compare  models  taken 
from  two  entirely  different  data  sets  (7:92).  However,  they 
do  not  recommend  using  the  Adjusted  R^  statistic  in  the 
latter  role. 

The  Adjusted  R^  statistic  (or  the  minimum  MSE  criterion) 
is  still  widely  used  in  practice. 

S^.  The  Sp  criterion,  originally  proposed  by 
Hocking  in  1976  (12:20),  has  considerable  appeal  and 
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consequently  receives  praise  in  recent  years.  The  Sp 
statistic  is  an  approximation  of  the  MSEP  based  solely  on 
the  data  and  number  of  variables-  As  is  the  case  with  MSEP, 
the  goal  of  this  criterion  is  to  find  the  minimum  value. 

SSE 

-  _  "31 

( n-p )  ( n-p-2 ) 

Breiman  and  Freedman  point  out  that  the  Sp  statistic  does 
not  necessary  provide  an  accurate  approximation  of  MSEP, 
but  works  none  the  less.  (3:132) 

The  advantages  of  this  method  are  numerous.  Looking  at 
Eq  (12)  gives  the  reader  an  idea  of  the  relative  ease  with 
which  S„  is  calculated.  What  makes  S.  even  more  appealing 
is  it  is  based  on  MSEP.  Thompson  points  out,  "This  method 
[Sp]  is  based  on  a  sound  criterion  -  that  of  minimizing  the 
expected  squared  distance  between  the  true  and  predicted 
values  of  the  dependent  variable,  y."  (27:6)  Since  Sp  is 

an  approximation  of  MSEP,  it  can  be  used  like  MSEP  to 
determine  the  optimal  number  of  regressors  to  include  in  the 
model .  (3:132) 

Sp  is  not  without  its  disadvantages.  It  must  be 
calculated  for  all  2^^  -1  possible  subsets.  (27:6)  Even 
though  it  requires  relatively  little  computation  effort,  it 
does  require  that  many  regressions  be  run.  Through  counter 
examples  Brieman  and  Freedman  show  that  when  true  variance 
due  to  prediction  equals  zero,  the  Sp  criterion  fails  to 


pick  the  optimal  number  of  variables  to  include  in  the 
model .  (3:132) 

Conclusion 

A  considerable  amount  of  literature  has  been  devoted  to 
linear  regression  in  the  last  decade.  This  review  has 
covered  a  few  of  the  techniques  mentioned  in  the  literature. 
By  no  means  should  this  review  be  considered  exhaustive. 
However,  it  is  intended  to  provide  an  excellent  review  of 
many  least  squares  regression  techniques  from  the  last 
decade . 

Since  the  calculations  required  for  the  "guaranteed"  all¬ 
subsets  method  virtually  make  it  impossible  to  use,  much  of 
the  effort  in  the  field  is  to  find  techniques  which  require 
far  less  calculation.  Unfortunately,  the  savings  in 
computer  time  is  off-set  by  the  results.  None  of  the 
techniques  mentioned  in  the  "cheap"  section  of  this  report 
guarantee  the  best-fitting  subset  of  regressors  will  be 
chosen . 

Biased  Regressors 

Up  to  this  point  the  discussion  of  literature  has  been 
restricted  to  selection  techniques  for  least  squares.  The 
advantage  of  least  squares  over  the  techniques  discussed  in 
this  section  is  that  the  estimators  are  best  linear  unbiased 
estimators,  or  BLUE.  However,  under  certain  circumstances 


the  estimators  from  least  squares  can  be  inferior  to  biased 
regressors. 

An  estimator  is  said  to  be  unbiased  if  its  expected  value 
of  the  estimator  is  the  parameter  value  itself. 

E(b)  =  B  (14) 

Conversely,  if  the  expected  value  of  the  estimator  is  not 
equal  to  the  parameter  value,  it  is  said  to  be  biased. 

The  following  techniques  are  intended  to  provide  the 
reader  with  some  insight  into  when  the  use  of  biased 
estimators  may  be  a  better  course  of  action  than  least 
squares.  Rather  than  discuss  several  biased  techniques, 
this  discussion  will  be  limited  to  two  methods  which  have 
received  a  great  deal  of  attention  in  recent  literature. 

Both  ridge  regression  and  principle  components,  the  two 
methods  to  be  discussed,  are  methods  for  dealing  with 
multicollinearity. 

Ridge  Regression.  There  has  been  a  significant  amount  of 
attention  given  to  Ridge  Regression  in  the  past  decade. 

Ridge  Regression  is  a  procedure  which  attempts  to  overcome 
the  effects  of  highly  correlated  data.  As  mentioned  before, 
many  of  the  least  squares  selection  criteria  perform  poorly 
under  such  conditions. 

There  are  two  situations  for  which  Draper  and  Sfnith 
"wholeheartedly  recommend  ridge  regression."  The  first 
situation  is  when  there  is  "a  Bayesian  formulation  of  a 
regression  problem  with  specific  prior  knowledge  of  a 
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(7:319)  The  advantage  over 


certain  type  on  the  parameters." 
least  squares  in  this  situation  is  that  the  B's  are 
restricted  a  priori.  The  second  situation  in  which  Draper 
and  Smith  recommend  Ridge  Regression  is  "a  formulation  of  a 
regression  problem  as  one  of  least  squares  subject  to  a 
specific  type  of  restriction  on  the  parameters."  (7:320) 
Draper  and  Smith  comment  that  under  the  two  previous 
situations  ridge  regression  is  absolutely  the  correct  model 
to  use.  However,  they  also  point  out  that  under  any  other 
situation,  there  needs  to  be  further  investigation  to 
determine  the  usefulness  of  ridge  regression. 

Dempster,  Schatzoff,  and  Wermuth  performed  a  simulation 
study  on  alternatives  to  least  squares  regression  in  1977. 
They  believe  ridge  regression  offers  a  drastic  improvement 
over  least  squares  regression  when  there  is  a  great  deal  of 
correlation  between  the  independent  variables  (5:77).  An 
interesting  comment  by  the  authors  is,  "From  a  frequentist 
standpoint,  it  has  long  been  recognized  that  good  mean 
squared  error  properties  do  not  necessarily  follow  from  the 
celebrated  minimum  variance  unbiasedness  properties  of  least 
squares,  since  in  certain  regions  of  the  parameter  space  the 
loss  from  increasing  the  squared  bias  can  be  overcompensated 
by  reducing  variance."  (5:77) 

In  a  subsequent  study  (19),  Ma)<.in  tested  the  ridge 
regression  procedure  and  found  it  performed  well.  Even 


though  his  models  were  fairly  trivial,  he  concluded  that 
ridge  regression  is,  in  fact,  a  good  technique. 


It  should  not,  however,  be  misconstrued  that  ridge 
regression  is  a  cure-all.  If  many  of  the  variables  under 
consideration  are  not  related  to  the  dependent  variable 
ridge  regression  does  not  perform  very  well  (13:375).  A 
suggestion  to  counter  this  drawback  is  to  delete  a  portion 
of  the  variables  if  there  is  reason  to  believe  many  have 
little  to  do  with  the  independent  variable.  Another  caveat 
of  using  ridge  regression  is  to  recognize  there  is  an  entire 
family  of  ridge  regression  techniques.  One  in  particular, 
REGF,  is  not  recommended  by  Dempster,  Schatzoff,  Wermuth  as 
a  practical  tool. 

Principle  Components.  Principle  components  is  another 
technique  used  to  combat  collinearity  and  has  received  a 
considerable  amount  of  attention  recently  .  Collinearity 
tends  to  yield  highly  unstable  estimates  because  the  design 
matrix  is  nearly  singular.  The  goal  of  principle  components 
is  similar  to  ridge  regression  in  the  sense  that  the  goal  is 
to  produce  estimates  which  are  not  greatly  effected  by 
collinear  data.  However,  unlike  ridge  regression  which 
assumes  nonsample  information  (15:909),  principle  components 
does  not  assume  any  nonsample  information  is  available. 

In  principle  components,  only  subsets  of  the  variables 
are  considered.  The  choice  of  subsets  can  be  made  "by 
economic  theory,  previous  statistical  results,  or  ad  hoc 
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dimensionality  reduction  procedures."  (15:909)  By  reducing 
the  number  of  variables  under  consideration,  principle 
components  attempts  to  eliminate  variables  which  do  not 
necessarily  contribute  to  the  model  and  cause  unnecessary 
col 1 inear i ty . 

There  is  some  criticism  of  the  method  of  principle 
components.  One  criticism  by  Judge,  Griffiths,  Hill, 
Lutkepohl,  and  Lee,  is  that  this  method  relies  on  the  data 
in  hand,  which  is  the  same  weaknesses  of  every  search 
method.  It  is  not  guaranteed  to  find  the  best  subset  of 
regressors  to  include  in  the  model  (15:910).  It  is  quite 
possible  that  the  subset  of  variables  chosen  does  not 
contain  all  of  the  true  regressors.  In  their  simulation 
study,  Hoerl,  Hoerl,  and  Schuenemeyer  demonstrated  that 
principle  components  performs  poorly.  (13:375)  "It  was 
rarely  noticeably  better  than  LS  [least  squares],  often 
worse  than  LS  and  uniformly  inferior  to  RRB  [a  ridge 
regression  technique]  for  all  criteria."  (13:375) 

Conclusion 

If  a  situation  arises  when  significant  mult icol 1 inearity 
exists,  and  it  is  not  appropriate  to  use  least  squares,  it 
may  be  appropriate  to  use  a  biased  regression  technique. 

Two  of  the  methods  which  have  received  recent  attention  are 
ridge  regression  and  principle  components.  However,  like 
least  squares  techniques,  these  techniques  give  guarantees 
for  finding  the  best  idel.  Also  like  least  squares,  both 


of  these  techniques  require  certain  conditions  to  perform 
optimally.  Ridge  regression  requires  some  type  of  nonsample 
information  while  principle  components  does  not.  However, 
principle  components  does  not  perform  nearly  as  well  as 
ridge  regression  and  even  least  squares.  These  trade-offs 
must  be  considered  when  deciding  on  a  technique  to  combat 
collinearity , 
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IV.  Methodology  and  Model  Development 


The  three  techniques  selected  for  this  research  were 
minimum  MSE,  minimum  ,  and  minimum  Sp. These  techniques 
were  chosen  for  the  following  reasons: 

(1)  Each  of  these  techniques  are  absolute  criterion.  That 
is,  they  is  no  "art"  necessary.  The  Sp,  and  MSE  statistic 
for  each  subset  of  variables  is  obtainable  under  some  all¬ 
subset  procedures,  such  as  PROC  on  SAS.  One  only  needs 
to  find  the  minimum  value  for  each  technique  and  the 
corresponding  set  of  variables  to  implement  these 
techniques.  Contrasted  with  the  Cp  close  to  p  technique 
which  leaves  confusion  as  to  whether  a  model  is  superior  to 
another  simply  on  the  basis  of  the  difference  between  Cp  and 
P* 

(2)  All  three  techniques  appear  in  the  last  decade's 
iiterature.  The  Minimum  MSE  procedure  used  to  be  one  of  the 
most  widely  used  methods.  Its  appeal  over  techniques  such 
as  Max  R^  stems  from  its  adjustment  for  degrees  of  freedom. 
More  recently,  Sp  seems  has  become  the  most  popular 
technique.  Its  appeal  is  based  on  the  principle  of 
minimizing  mean  square  errors  of  prediction. 

(3)  The  Cp  criterion  is  also  based  on  MSEP,  and  some  of 
authors  praise  this  criterion. 
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Overview 


With  any  procedure,  certain  assumptions  must  hold  to 
insure  proper  execution  and  least  squares  is  no  exception. 

As  mentioned,  by  using  least  squares  one  assumes  the  data  is 
typical  of  the  population,  and  the  error  terms  are 
independent  and  identically  distributed  from  a  normal 
population  with  an  expected  value  of  zero  and  a  constant 
variance  o^.  Another  assumption  often  made,  yet  not 
entirely  justified,  is  that  all  the  variables  collected 
relate  to  the  dependent  variable. 

Objective 

The  goal  of  this  thesis  is  to  determine  the  behavior  of 
the  Min  MSE,  Min  C„,  and  Min  S,,  variable  selection  criteria. 
To  do  this,  response  surface  techniques  was  implemented. 
Response  Surface  Methodology 

When  selecting  variables  to  include  in  the  original 
variable  pool  for  the  least  squares  regression  equation,  it 
is  desirable  to  select  only  those  variables  which 
significantly  contribute  to  the  model.  However,  there  is  no 
guaranteed  method  to  screen  the  extraneous  variables ( random 
noise  terms  which  do  not  contribute  at  all  to  the  model) 
from  the  variable  pool.  Even  more  discouraging,  once  in  the 
variable  pool  there  is  no  criterion  which  guarantees  that  no 
extraneous  variables  will  be  chosen  for  the  model.  The 
problem  of  selecting  extraneous  variables  is  not  restricted 
to  the  "cheap"  methods.  In  fact,  one  of  Miller's  guaranteed 
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methods,  the  all-subset  procedure,  which  is  suppose  to 
perform  quite  well,  also  occasionally  chooses  extraneous 
variables.  The  limitations  of  variable  selection  criteria, 
including  the  "guaranteed"  techniques  will  be  emphasized 
since  all  three  of  the  criteria  under  consideration  are  all¬ 
subset  procedures. 

Since  there  really  are  no  "guaranteed  methods"  for 
capturing  all  and  only  the  true  variables,  it  is 
advantageous  to  strive  for  the  highest  percentage  of  those 
variables  chosen  correctly.  Therefore  the  performance 
measure  for  the  Response  Surface  is  the  following: 


PM  = 


(number  of  correct  variables  chosen) 


(  number  chosen) 


(15) 


PM  is  the  logical  choice  for  two  reasons.  First,  the  best 
model  may  not  include  all  variables  it  is  generated  from, 
but  only  the  most  significant.  Even  though  an  independent 
variable  may  have  been  generated  from  ten  variables,  the 
best  model  of  reality  using,  say  MSE,  may  only  contain  two 
of  those  variables.  Therefore,  PM  compensates  by 
determining  the  percentage  of  correct  variables  chosen. 
Second,  PM  is  influenced  by  the  number  of  extraneous 
variables  chosen.  If  only  two  variables  are  chosen,  and  one 
of  them  is  an  extraneous  variable,  it  is  worse  than  when  ten 


variables  are  chosen  and  only  one  of  them  is  extraneous.  PM 
allows  for  such  an  adjustment. 

By  using  Response  Surface  Methodology,  equations  made  up 
of  significant  factors  ( multicollinearity ,  sample  size, 
etc.)  for  each  of  the  three  criteria  can  be  fitted.  Using 
these  equations,  a  comparison  of  all  three  criteria  under 
best-case  and  worst-case  scenarios  can  be  completed. 

Data  Generation  For  Response  Surface.  The  data  for  this 
study  was  generated  from  the  following  equation: 

Y,  =  X.,  +  +  X3,  +  X^,  +  e,  (16) 

where 


e. 


is  the  dependent  variable, 
are  correlated,  randomly  generated 
dependent  variables, 
is  a  noise  term  to  create  variance 
in  the  model. 


Most  simulation  studies  investigate  variable  selection 
techniques  with  all  relevant  dependent  variables  plus  some 
extraneous  variables  included  in  the  variable  pool.  This 
study  attempts  to  show  what  happens  when  one  of  the  relevant 
variables  is  not  included  in  the  variable  pool. 

After  the  data  is  created  from  equation  (16),  the 
independent  variable,  X^,  is  dropped  from  consideration. 

This  simulates  the  situation  which  arises  when  a  significant 
variable  is  not  included  in  the  variable  pool.  In  addition 
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either  one  or  three  noise  variables  are  included  in  the 
variable  pool  to  simulate  data  collection  on  extraneous 
variables . 

Factors .  An  equation  made  up  of  significant  factors  and 
factor  interactions  which  adequately  predicts  the  usefulness 
of  each  of  the  three  techniques  must  be  found.  Ideally,  the 
significant  factors  are  observable  a  priori  (e.g.  sample 
size,  correlation,  etc.),  thus,  allowing  for  compensation 
prior  to  use.  The  following  are  the  factors  for  this  study: 

(1)  The  number  of  extraneous  variables  (  EX^ ,  EX2,EXj)  in 
the  original  variable  pool.  These  variables  are  noise,  and 
therefore  are  theoretically  independent  from  the  dependent 
variable.  In  this  study,  at  the  low  setting  the  number  of 
extraneous  variables  is  1 ,  and  3  for  the  high  setting. 

(2)  The  amount  of  the  correlation  between  the  dependent 
variables,  X.,  X^,  Xv,  and  X^.  The  low  setting  is 
orthogonal,  or  zero  correlation,  while  the  high  setting  is 
0.9,  which  is  highly  correlated. 

(3)  The  variance  of  the  extraneous  variables.  The  low 
setting  for  the  variance  is  1,  and  the  high  setting  is  100. 

(4)  The  variance  of  the  independent  variables.  The  low 
setting  for  the  variance  is  1,  and  the  high  setting  is  100. 

(5)  The  sample  size.  The  low  setting  for  the  sample  size 
is  deliberately  near  the  threshold  of  the  usefulness  of  the 
Sp  criterion.  A  requirement  of  the  Sp  statistic  is  (n-p-2) 
should  be  greater  than  zero.  At  the  low  setting  the  sample 
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size  is  10.  The  maximum  value  for  p  is  7,  in  the  three 
extraneous  variable  case).  When  n  equals  10  and  p  equals  7, 
the  denominator  of  the  statistic  is  one.  Therefore,  the 
threshold  for  sample  size  is  10.  If  the  sample  size  were 
any  smaller,  Sp  could  not  be  found  for  all  possible  subsets. 
Thus,  the  low  setting  for  sample  size  is  10,  while  the  high 
setting  is  20. 

(6)  The  variance  of  the  €  term.  The  low  setting  for  the 
variance  of  €  is  0.0625,  and  the  high  setting  is  0.25.  The 
computer  program  used  to  create  the  correlated  data  can  be 
found  in  Appendix  B. 

When  using  Response  Surface  Methodology  it  is  convenient 
to  work  with  coded  factors  (-1,  1  variables)  for  the 

following  reasons: 

(1)  By  coding  the  factors,  the  resulting  variables  are 
of  the  same  magnitude. 

(2)  The  calculations  necessary  for  the  estimates  are 
simpl i f ied . 

(3)  The  resulting  design  matrix,  Z,  is  orthogonal.  As 
a  result,  the  stepwise  procedure  can  be  used  to  find  the 
significant  factors  with  confidence. 

The  following  equations  are  necessary  to  code  the 
variables : 
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=  number  of  extraneous  vars  -  2 
Zj  =  (correlation  -  0.45)  /  0.45 

Zj  =  (variance  of  extraneous  vars  -  50.5)  /  49.5 
Z^  =  (variance  of  independent  vars-  50.5)  /  49.5 
Zj  =  (sample  size  -  15)  /  5 

Z^  =  (variance  of  6  term  -  0.15625)  /  0.09375 
where  Z^,...,Z^  are  the  coded  variables. 

Table  I. 

Variable  Coding  for  Response  Surface  Methodology 

Non-Coded  Coded 


Variable 

Low 

High 

Variable 

Low 

High 

Number  of 

extraneous  vars 

1  .0 

3.0 

Zl 

-1 

1 

Correlation 

o 

• 

o 

0.9 

Za 

-1 

1 

Variance  of 
extraneous  vars 

1 .0 

100.0 

Z5 

-1 

1 

Variance  of 
independent  vars 

1  .0 

100.0 

Z4 

-1 

1 

Sample  Size 

10.0 

20.0 

Z5 

-1 

1 

variance  of 
€  term 

0.0625 

0.25 

Z6 

-1 

1 

Experiment .  It  seems  reasonable  to  assume 
significance  in  interactions  between  factors.  For  example, 
there  might  be  a  significant  interaction  between  two  of  the 
main  factors  mentioned  above.  To  insure  the  accurate 


calculation  of  estimates  for  both  the  main  factors  and 


factor  interactions,  a  full  2"  factorial  design  is 
necessary.  To  construct  the  design  matrix  for  a  full 
factorial  design,  the  main  factors  are  varied  from  their  low 
settings  to  their  high  settings  in  binary  fashion.  The 
interaction  terms  are  simply  the  product  of  the 

corresponding  main  factors.  An  example  of  this  process 

2 

using  a  2"  full  factorial  design  i s  summa r ized  in  the  table 
below. 


Table  II. 

Example  of  Coding  Interaction  Variables 


Zl 

ZiZ^  (interaction) 

-1 

-1 

1 

-1 

1 

-1 

1 

-1 

-1 

1 

1 

1 

If  a  design  with  less  than  2°  runs  is  used,  information  on 
some  of  the  interactions  would  be  unobtainable. 

Each  of  the  2^  runs  contains  60  replications.  The  raw 
data  was  obtained  by  running  regressions  on  unprocessed  data 
sets  from  Appendix  B.  The  code  necessary  to  perform  the 
regression  analysis  is  found  in  Appendix  C.  A  FORTRAN 
program  was  written  to  accumulate  the  statistics.  The 
program  and  a  sample  output  can  be  found  in  Appendix  D  and  E 
respectively.  The  following  statistics  were  collected  for 
each  of  the  64  runs. 
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(  1 )  The  average  number  of  variables  chosen  using  each 


technique . 

(2)  The  average  number  of  correct  and  extraneous 
variables  of  those  chosen  by  the  technique. 

Results 

Since  the  design  matrix  for  this  experiment  is 
orthogonal,  stepwise  regression  is  used  to  select  the 
significant  factors.  Resulting  equations  strictly  screen 
factors  which  are  most  significant  to  their  respective 
response.  The  equations  should  not  be  used  for  predicting 
the  percentage  of  correct  variables  chosen.  Several 
unsuccessful  transformations  were  attempted  to  obtain  a 
reasonable  model,  however  none  made  common  sense.  Many 
models  appeared  to  fit  well  using  several  common  criteria 
(R',  Adj  R^,  residual  plots,  etc),  however,  the  models  were 
not  logical.  Using  stepwise  regression  with  a  significance 
level  of  0.01,  yielded  the  equations  given  below.  Again, 
the  role  of  these  equations  is  restricted  to  determining  the 
most  significant  factors  for  a  comparison  of  the  three 
criteria . 

Min  MSE.  For  the  Minimum  MSE  case,  only  two  main  factors 
were  significant.  They  are  the  number  of  extraneous 
variables  in  the  variable  pool  and  the  sample  size.  Since 
the  variables  are  coded  and  the  values  of  the  standard 
errors  for  all  the  estimates  are  equal,  only  one  value  is 
printed  for  the  standard  errors. 
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=  .9411 
Adj  =  .9392 

ymse(9t  correct)  =  .64448797  -  .  1  20997  (#  ex) 

+  . 0 1 04 1 5 ( sample ) 

standard  errors  =  .0039 
where 

#  ex  is  the  number  of  extraneous  variables  allowed 
in  the  variable  pool. 

sample  is  sample  size  of  each  replication  for  a 
given  run. 

This  equation  provides  insight  into  the  usefulness  of  the 
Min  MSE  criterion.  Within  the  data  region,  the  effect  of 
increasing  the  sample  size  can  be  observed.  For  every 
"extra"  data  point,  an  increase  of  1.04  percent  of  correct 
variables  chosen  is  expected.  This  is  useful  information 
since  sample  size  can  be  moderately  controlled.  The 
equation  also  indicates  a  decrease  of  12  percent  of  correct 
variables  chosen  for  every  extraneous  variable  allowed  into 
the  variable  pool.  At  first  this  information  seems 
irrelevant  since  there  is  no  way  of  telling  a  priori  if  a 
variable  is  relevant.  However,  it  does  emphasize  the  need 
to  screen  the  variable  pool  before  using  the  Min  MSE 
criterion.  It  also  shows  the  danger  of  the  )ritchen  sinlc 
approach.  The  Jcitchen  sink  approach  is  where  every  possible 
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variable  is  tossed  into  the  variable  pool  and  the  computer 
is  allowed  to  make  all  the  decisions.  Therefore,  the  two 
significant  main  factors  using  the  Min  MSE  criterion  are 
sample  size  and  the  number  of  extraneous  variables,  and  the 
number  of  extraneous  variables  is  the  most  significant. 

Min  S„.  For  the  Sp  case,  two  main  factors  and  one  three- 
way  interaction  are  significant. 


R 

.8811 

Adj 

.  8751 

ysp(^  correct) 

. 65598 

-.1148(#  ex)  -.0184  (6  var) 

-.0168(int) 

standard  errors  = 

.  0056 

where  #  ex  is  number  of  extraneous  variables, 
e  var  is  variance  of  the  €  term, 
int  is  the  three-way  interaction  term. 

The  three  terms  are:  number  of  extraneous 
variables,  the  €  variance  and  the 
variance  of  the  X  variables. 

The  information  obtained  here  is  acceptable  but  not  quite 
as  useful  as  in  the  MSE  case.  Apparently  the  most 
significant  factors  which  effect  Sp  within  the  observed  data 
space  IS  the  number  of  extraneous  variables  and  the 
variation  of  Y.  Again,  the  point  to  emphasize  is  as  few 
extraneous  variables  as  possible  should  be  allowed  in  the 
variable  pool.  It  is  interesting  to  note  that  sample  size 
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is  insignif leant  at  the  a=.01  level.  The  low  level  set  for 
sample  size  was  near  the  threshold  of  Sp's  usefulness  on 
purpose.  Consequently,  Sp  might  be  a  good  criterion  to  use 
when  the  sample  size  is  small. 

Min  Cp.  Unfortunately  the  resulting  equation  for  the 
case  is  not  as  descriptive  as  the  two  previous  cases.  The 
Cp  equation  included  two  main  factors,  one  three  way 
interaction,  two  four-way  interactions,  and  a  five-way 
interaction. 

=  .9280 

Adj  =  .9202 

yep (^correct )  =  .65  -.11(#ex)  -  .01  (6var) 

-.01  ( #ex , xvar , €var ) 
+.02(cor,xvar,exvar,€var) 

+  . 0 1 ( #ex , cor , xvar , exvar , Gvar ) 

standard  error  =  .0045 
where 

#ex  is  the  number  of  extraneous  variables  in  the 
variable  pool. 

cor  is  the  amount  of  correlation  between  the 
independent  variables. 

exvar  is  the  variance  of  the  extraneous  variables, 

xvar  is  the  variance  of  the  independent  variables. 

€  var  is  the  variance  of  the  6  term. 

Only  in  this  equation  is  the  amount  of  correlation  among  the 
X's  significant.  Even  though  it  appears  in  the  equation,  it 
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is  not  one  of  the  two  main  factors,  but  does  appear  in  the 
four  and  five  way  interaction  terms. 

As  is  the  case  with  the  MSE  equation  and  the  Sp  equation, 
the  number  of  extraneous  variables  included  in  the  variable 
pool  is  the  most  significant  factor.  However,  probably  the 
most  notable  result  of  this  experiment  is  the  repetition  of 
variance  factors  appearing  in  the  interactions.  Actually 
this  is  expected,  since  the  C.  statistic  is  heavily 
dependent  on  the  total  variation  of  the  variable  pool. 
Comparison  of  Techniques 

Up  to  this  point,  analysis  has  bet n  limited  to  intra¬ 
technique.  That  is  to  say,  the  models  constructed  in  the 
Response  Surface  phase  show  the  effect  of  varying  factors 
u;  ing  a  specific  technique.  Determining  which  technique 
actually  outperforms  the  other  two  techniques  is  more 
useful . 

Choice  of  a  Comparison  Criterion.  Deciding  which 
performance  measure  to  use  for  comparing  the  Min  MSE,  Min 
C-,  and  Min  S,,  variable  selection  criteria  is  not  trivial. 

Due  to  the  mechanics  of  the  Min  MSE  procedure,  the  resulting 
subset  of  variables  is  always  a  super-set  of  both  the  Min  Cp 
and  Min  Sp  procedures.  Also,  in  each  of  the  sixty-four 
replications,  the  Min  MSE  procedure  picked  the  highest 
percentage  of  correct  variables  of  those  chosen; 
additionally  it  picked  the  most  correct  variables  in 
absolute  terms.  It  seems  the  Min  MSE  criterion  is  superior 
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to  the  other  two  criterion.  However,  these  statistics  are  a 
bit  misleading. 

In  each  of  the  sixty-four  replications,  the  Min  MSB 
criterion  chose  the  most  extraneous  variables  in  absolute 
terms.  Remembering  the  concept  of  best  model  versus  true 
model,  it  is  probably  worse  to  have  more  extraneous 
variables  in  the  model  than  the  highest  percentage  of 
correct  variables  of  those  chosen.  Therefore,  comparing 
percentages  to  determine  the  "best  technique"  is  not  used. 

Since  there  is  no  clear-cut  criterion  for  comparing 
variable  selection  techniques,  it  was  decided  to  compare  how 
well  each  criterion  performs  using  theoretical  minimum  mean 
square  error  of  prediction  (TMSEP). 

Z  (Y.  -  Y  )’ 

TMSEP  =  ' _  (17) 

(n  -  p) 

where 

Yj  is  the  theoretical  value  of  the  independent 
variable  Y. 

Yjj  is  the  predicted  value  of  Y,  using  one  of  the  three 
criteria . 

n  is  the  sample  size. 

p  is  the  number  of  parameters  in  the  predicting 
equation. 
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There  are  two  reasons  for  picking  TMSEP  as  the 
performance  measure.  First,  MSEP  has  received  considerable 
attention  during  the  past  decade  as  the  most  promising 
criterion  for  variable  selection.  Second,  the  Sp  criterion, 
which  is  based  on  MSEP,  is  praised  and  considered  by  many  as 
the  method  which  insures  the  model  selected  has  the  Min 
MSEP. 

The  TMSEP  criterion  is  a  variation  of  MSEP.  Like  MSEP, 
TMSEP  calculates  the  squared  difference  between  the 
predicted  value  and  the  actual  value  of  the  independent 
variable  and  adjusts  the  value  for  degrees  of  freedom. 
However,  TMSEP  differs  from  MSEP  in  its  calculation.  TMSEP 
is  the  squared  difference  between  the  actual  value,  from  the 
underlying  equation  that  is  generating  the  independent 
variable  (not  including  the  error  term,  €),  and  the 
predicted  value  using  the  model  picked  by  the  variable 
selection  procedure.  The  resulting  value  is  the  theoretical 
mean  square  error  of  prediction  (TMSEP). 

At  first,  it  appears  this  performance  measure  unfairly 
favors  the  Minimum  and  Minimum  S-  criteria.  Both  of 
these  criteria  are  based  on  minimum  MSEP.  The  difference 
between  the  two  lies  in  the  assumption  that  the  regressors 
are  fixed  in  the  Cp  case  and  are  random  in  the  Sp  case. 

Since  the  regressors  in  this  study  are  randomly  generated, 
it  is  then  assumed  that  the  Sp  criterion  would  outperform 
the  other  two.  However,  this  is  not  necessarily  true. 


It  is  assumed  when  calculating  the  Sp  and  Cp  statistics 
that  all  relevant  variables  are  included  in  the  variable 
pool.  It  is  also  assumed  that  variable  pool  does  not 
contain  extraneous  variables.  In  this  study  both  of  these 
assumptions  are  violated.  Therefore,  it  is  possible  that 
either  the  or  MSE  criterion  could  outperform  the  Sp 
criterion.  Since  the  MSE  criterion  is  not  designed  to 
minimize  MSEP,  it  is  doubtful  whether  it  would  outperform 
the  other  two  criteria  using  TMSEP  as  the  performance 
measure. 

Performance  of  the  Selection  Technique  using  TMSEP. 
Measurement  of  each  criterion’s  performance  is  recorded 
under  best-case  and  worst-case  factor  settings.  If  a 
technique  outperformed  the  other  two  in  both  the  best-case 
scenario  and  worst-case  scenario,  it  is  superior  to  the 
other  two. 

Since  determining  the  necessary  sample  size  is  difficult, 
runs  were  made  until  the  distributions  for  each  technique 
stabilized.  The  number  of  necessary  number  of  runs  was  180 
for  both  the  best-case  and  worst-case  scenarios. 

Method  of  Comparison.  For  each  of  the  360  runs,  the 
TMSEP  value  was  calculated  for  every  subset  of  variables. 
For  example,  when  the  factor  setting  for  extraneous 
variables  was  one  (best-case  factor  setting),  there  were 
four  possible  regressors.  Since  there  were  2*^-1  subsets 
possible,  in  our  example,  fifteen  TMSEP  values  were 
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calculated.  In  the  three  extraneous  variable  case  the 
number  of  values  increases  to  sixty-three.  The  2  -1  TMSEP 
values  were  then  ordered  from  lowest  to  highest;  lowest 
being  more  desirable.  The  TMSEP  values  are  then  calculated 
for  the  models  chosen  by  each  of  the  three  techniques. 

Then,  the  values  calculated  from  each  technique's  choice  was 
compared  to  the  rank  ordered  values  and  its  ranking  was 
recorded.  For  instance,  if  the  value  chosen  by  a  technique 
was  identical  to  the  lowest  value  of  TMSEPs,  then  a  "one" 
was  recorded.  Additionally,  the  ranking  of  the  TMSEP  value 
for  each  technique  was  compared  to  the  other  two  and 
recorded.  A  FORTRAN  program  was  written  to  execute  the 
procedure  outlined  in  this  section.  The  code  for  this 
program  can  be  found  in  Appendix  F,  and  a  sample  output  can 


be 

the 
obt 
f  ol 


found  in  Appendix  G. 

Best-Case  Comparison.  Using 
Response  Surface  phase,  a  common 
ained  which  constitutes  a  best-ca 
lowing  settings  maximize  each  tec 


the  equations  found 
set  of  factors  was 
se  scenario.  The 
hniques  performance: 


in 
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Table  III. 

Factor  Settings  For  Best-Case  Scenario 


variable  Non-Coded  value  coded  value 


Number  of 

extraneous  variables 

1 .0 

-1  .0 

Correlation 

0.9 

+  1 . 0 

Variance 

of  extraneous  variables 

100.0 

+  1 . 0 

Variance 

of  independent  variables 

1 . 0 

-1.0 

Sample  size 

20.0 

+  1 . 0 

Variance  of  e  term 

0.0625 

-1 .0 

The  Min  S^  criterion  was  derived  under  the  assumption  that 
the  regressors  are  random,  while  the  Min  Cp  criterion  was 
derived  under  the  assumption  that  the  regressors  are  fixed. 
Since  the  regressors  are  random  in  this  experiment,  it  was 
expected  that  there  would  be  a  significant  difference 
between  the  two.  The  following  two  figures  summarize  the 
results  from  the  best-case  comparison. 
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Figure  2.  Bar  Graph  for  Best-Case  Comparison  Results 


Figure  3.  Cumulative  Distribution  Function  for  Best-Case 
Comparison 
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The  cumulative  distribution  function  of  each  technique 
indicates  there  probably  is  a  significant  difference  between 
the  two  MSEP  procedures,  the  difference  between  MSE  and  the 


other  two  is  obvious.  The  chart  below  summarizes  the  number 
of  times  each  technique  chose  the  lowest  TMSEP  value  amongst 
the  three. 

Table  IV. 

Best-Case  Scenario  Results 


Criteria 

Number  of  times 
Percentage 

Lowest  Value 

Cp  only 

1 

0.5^ 

So  only 

9 

5.0% 

MSE  only 

1 

0.5% 

Cp  and  Sp  tied 

58 

3  2.0% 

Cp,  Sp,  and  MSE  tied 

1  1  1 

62.0% 

Allowing  for  ties,  the  fol 
each  technique  found  the 
three . 


lowing 

lowest 


are  the  percentages  that 
TMSEP  value  amongst  the 


Table  V. 

Best-Case  Scenario  Results 


Criterion 

Percentage  of  time 
lowest,  or  tied  for 
lowest  amongst  the 
three 

Cp 

94.0% 

Sp 

99.0% 

MSE 

62.0% 
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The  table  above  reaffirms  our  expectation  that  the  Min  Sp 
criterion  is  superior  to  the  Min  criterion  under  a  best- 
case  scenario.  To  determine  if  the  difference  actually 
exists  a  hypothesis  test  is  performed.  The  null  hypothesis, 
Ho,  is:  the  percentage  of  lowest  TMSEP  values  chosen  solely 
by  the  Min  Sp  criterion  equals  the  percentage  chosen  solely 
by  the  Min  Cp  criterion.  The  alternative  hypothesis,  Ha, 
is:  the  percentage  chosen  solely  by  the  Min  Sp  criterion  is 
not  equal  to  the  percentage  chosen  solely  by  the  Min  Cp 
criterion. 


Ho:  Pep  =  Psp 
Ha:  Pep  ^  Psp 


a  =  0.05 

critical  probability  =  0.039 


The  following  is  the  equation  used  to  calculate  the  critical 
probability : 
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where  i  is  number  of  occurrences  out  of  9  trials,  that  Cp 
chose  a  model  with  a  lower  TMSEP  value. 

9  is  the  total  number  of  times  the  two  MSEP 
criteria  differed. 

This  equation  is  based  on  the  binomial  distribution  with  a 
probability  of  0.5.  That  is,  the  chance  of  one  criterion 
picking  the  lowest  TMSEP  value  is  equal  to  the  other. 

Since  the  significance  level  is  larger  than  the  critical 
probability,  the  null  hypothesis  is  rejected.  The 
conclusion  is  there  is  a  significant  difference  between  the 
two  criteria  under  the  a  best-case  scenario,  and  the  Min  Sp 
is  superior. 

Worst-Case  Comparison.  By  reversing  the  factor  settings 
of  the  best-case  scenario,  a  worst-case  scenario  is 
obtained. 
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Table  VI. 

Factor  Settings  For  Worst-Case  Scenario 


variable  Non 

-Coded  value 

coded  value 

Number  of 

extraneous  variables 

3.0 

+  1 . 0 

Correlation 

o 

« 

o 

-1 .0 

Variance 

of  extraneous  variables 

1  .  0 

-1  .0 

Variance 

of  independent  variables 

100.0 

+  1 . 0 

Sampe  size 

10.0 

-1.0 

Variance  of  €  term 

0.25 

+ 

• 

o 

Similar  to  the  best-case 

scenario  results, 

the  Min  MSE 

criterion  did  significantly  worse  than  the  other  two 
technique  .  The  graph  below  shows  the  cumulative 
distribution  function  for  each  of  the  criteria  (the  worst 
case  bar  graphs  are  located  in  Appendix  A). 
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Notice  the  cumulative  distribution  for  Cp  is  larger  than  the 
other  two  criteria  for  most  rank  ordered  TMSEP  values-  This 
is  surprising  since  the  factors  are  random.  The  question  is 
whether  the  difference  between  the  two  MSEP  criteria  is 
significant.  If  so,  the  usefulness  of  the  minimum  Sp 
criterion  would  be  questionable. 


Table  VII. 

Worst-Case  Scenario  Results 


Criteria  Number  of  times  Percentage 

Lowest  Value 


C_  only 

28 

16.0?i 

Sp  only 

4 

2 . 0% 

MSE  only 

0 

0.0% 

C.  and  Sp  tied 

81 

45.0% 

C_,  S_,  and  MSE  tied 

67 

37.0% 

Allowing  for  ties,  the  percentage  each  technique  found  the 
lowest  TMSEP  value  amongst  the  three  are  summarized  in  the 
•  table  below. 


Table  VIII. 


Worst-Case 

Scenario  Results 

• 

Criterion 

Percentage  of  time 
lowest,  or  tied  for 
lowest  amongst  the 
three 

• 

Cp 

98.0% 

Sp 

91 .0% 

MSE 

67.0% 
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Since  the  Min  Cp  criterion  appears  to  be  significantly 
better  than  the  Min  Sp  criterion,  a  two  tailed  hypothesis 
test  at  the  a=0.05  level  was  performed.  This  test  was 
identical  to  the  test  in  the  best  case  scenario.  The  null 
hypothesis  was:  the  percentage  of  lowest  TMSEP  values  chosen 
solely  by  the  Min  Sp  criterion  is  equal  to  the  percentage 
chosen  by  the  Min  Cp  criterion.  While  the  alternative 
hypothesis  was:  the  percentage  chosen  by  the  Min  Sp 
criterion  is  not  equal  to  the  percentage  chosen  by  the  Min 
Cp  criterion. 


Ho:  Pep  =  Psp 
Ha:  Pep  4  Psp 


a  =  0.05 

critical  probability  =  0.00002 


The  critical  probability  for  this  hypothesis  test  was 
calculated  in  exactly  the  same  manner  as  in  the  best-case 
scenario.  The  equation  necessary  to  find  the  critical 
probability  is: 
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where 


i  is  number  of  occurrences  out  of  32  trials, 
Cp  chose  a  model  with  a  lower  TMSEP  value 
32  is  the  total  number  of  times  the  two  MSEP 
criteria  differed. 


Since  the  critical  probability  is  less  than  the 
significance  level,  the  null  hypothesis  is  rejected, 
the  conclusion  is  the  Min  Cp  criterion  significantly 


that 


Thus  , 


outperforms  the  Min  Sp  criterion  in  the  worst-case  scenario. 


V.  Conclusions  and  Further  Research 

The  objectives  of  this  research  were:  (1)  identify  some 
promising  least  squares  selection  procedures  discussed  in 
the  literature  during  the  previous  decade,  (2)  use  Response 
Surface  Methodology  to  find  the  factors  which  most 
significantly  drive  the  selection  process  for  each  of  the 
these  techniques,  and  (3)  make  a  best-case  and  worst-case 
scenario  comparison  of  the  techniques. 

Conclusion 

The  three  techniques  chosen,  Min  MSB,  Min  C^,  and  Min  S^, 
have  received  praise  in  the  last  decade's  literature.  About 
ten  years  ago  Min  MSB  was  considered  a  favorable  technique 
because  of  its  similarity  to  the  Max  criterion,  with  an 
adjustment  for  degrees  of  freedom.  In  resent  literature, 
the  Min  and  Min  C,  criteria  have  received  the  majority  of 
praise.  Both  techniques  are  based  on  the  principle  of 
minimizing  the  mean  square  errors  of  prediction.  Of  the 
two,  the  Min  criterion  has  received  more  attention.  Its 
uses  vary  from  finding  the  optimal  number  of  variables  to 
include  in  the  model,  to  finding  the  best  subset  of 
variables.  Unlike  Min  C„,  Min  S,,  is  designed  for  random 
regressors,  which  is  more  practical. 

During  the  Response  Surface  phase,  emphasis  was  placed  on 
the  technique's  performance  under  varying  degrees  of 
multicollinearity ,  variable  variance,  number  of  variables. 
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and  sample  size.  A  full  2^  factorial  design  was  constructed 
to  find  the  most  significant  factors  and  their  interactions. 
The  three  equations  developed  from  the  Response  Surface 
phase  show  the  relative  weights  of  the  most  significant 
variables.  These  equations  serve  as  a  screening  method,  and 
should  not  be  used  to  predict  the  percentage  of  correct 
variables  of  those  chosen  for  any  of  the  techniques. 

The  most  noticeable  result  from  the  Response  Surface 
phase  dealt  with  the  number  of  extraneous  variables  which 
were  included  in  the  variable  pool.  This  factor  was 
overwhelmingly  the  most  significant  in  determining  the 
percentage  of  correct  variables  chosen  by  a  technique.  In 
fact,  if  the  number  of  extraneous  variables  was  the  only 
variable  in  any  of  the  Response  Surface  equations,  it  would 
account  for  over  85%  of  the  variation  about  the  mean. 
Therefore,  it  is  very  important  to  screen  the  variables  used 
in  a  regression  analysis  by  some  other  means  than  just 
letting  a  computer  program  do  the  work. 

The  conclusion  dealing  with  extraneous  variables  is 
consistent  with  Freedman  (9),  Flack  and  Chang  (8),  and 
Hoerl,  Hoerl,  Schuenemeyer  (13),  as  well  as  Miller's 
suggested  technique  for  a  stepwise  regression  stopping-rule. 
It  can  not  be  over-emphasized  that  the  "kitchen  sink" 
approach  is  not  the  best  way  to  go.  One  must  decide  which 
variables  seem  relevant  to  the  dependent  variable.  If  a 
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good  reason  for  including  a  variable  can  not  be  determined, 
then  it  should  not  be  included  in  the  model. 

Using  common  factors  of  the  three  equations  found  in  the 
Response  Surface  phase,  a  best-case  and  worst-case  scenario 
comparison  was  made  using  the  theoretical  mean  square  errors 
of  prediction  (TMSEP).  Since  the  information  necessary  to 
find  TMSEP  is  not  available  in  practice,  this  statistic's 
usefulness  is  restricted  to  the  simulation  arena.  However, 

®  technique  such  as  (Min  S, )  is  suppose  to  minimize  MSEP, 
then  it  is  expected  that  it  will  also  have  one  of  the  lowest 
^  TMSEPs  of  all  possible  models.  Likewise,  since  Min  is 
also  based  on  minimizing  MSEP,  it  was  expected  to  do  fairly 
well. 

The  results  of  the  best-case  comparison  indicate  the  Min 
S„  criterion  performs  better  than  Min  C^.  Since  the  Min  S,, 
t^^^terion  is  by  definition  made  for  random  regressors,  one 
would  expect  it  to  do  significantly  better  than  Min  Cp. 

Both  criteria  did  very  well  in  absolute  terms.  Over  75^  of 
the  TMSEP  values  calculated  from  the  models  chosen  by  Sp  and 
Cp  were  in  the  upper  half  of  rank  ordered  TMSEP  values.  14% 
of  the  values  found  from  the  MSEP  criteria's  model  were  the 
absolute  lowest  TMSEP  value  possible.  This  is  extremely 
good  considering  a  fundamental  regressor  was  dropped  from 
the  variable  pool  prior  to  using  the  variable  selection 
criterion. 
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The  results  from  the  worst-case  comparison  were  shocking. 


Surprisingly,  Min  outperformed  Sp.  What  makes  this  so 
surprising  is  the  assumption  that  the  regressors  must  be 
fixed  to  implement  Cp.  The  difference  between  the  Cp  and  Sp 
could  be  attributed  to  many  things.  First,  when  these  two 
criteria  were  derived  there  was  an  assumption  that  all 
relevant  regressors  were  included,  and  no  extraneous 
variables  were  included  in  the  variable  pool.  Both  of  these 
assumptions  have  been  violated  in  this  research.  Under 
these  circumstances,  it  is  possible  Cp  outperforms  Sp 
regardless  of  the  nature  of  the  regressors.  In  the  best- 
case  scenario,  the  MSE  criterion  performed  significantly 
worse  than  the  other  two  criteria. 

The  results  of  this  thesis  provide  insight  into  the 
usefulness  of  the  Min  MSE,  Min  Sp,  and  Min  Cp  criteria 
under  realistic  conditions.  It  is  possible  that  variables 
which  significantly  contribute  to  a  model  may  be  left  out  of 
the  variable  pool.  It  is  also  possible  some  extraneous 
variables  will  be  included.  The  most  important  lesson  of 
this  research  is:  the  number  of  extraneous  variables  which 
are  in  the  variable  pool  significantly  detriments  the 
selection  process.  There  must  be  some  thought,  commonly 
referred  to  by  Operation  Research  analysts  as  the  "art", 
used  when  picking  variables  to  include  in  the  variable  pool. 
Not  even  the  state-of-the-art  variable  selection  criteria 
are  able  to  perform  well  when  the  variables  have  not  been 
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properly  screened  before  implementation.  This  information 
may  be  useful  to  those  who  make  a  practice  of  collecting 
data  on  everything  and  letting  the  computer  pick  the 
"optimal  model." 

Another  useful  result  of  this  research  is  the  comparison 
of  two  MSEP  criteria.  A  great  deal  of  praise  has  been  given 
to  the  minimum  Sp  criterion  in  the  past  decade.  It  has  been 
identified  as  "one  of  the  most  promising"  when  the 
regressors  are  random,  and  one  desires  to  minimize  the  mean 
square  error  of 

prediction.  The  minimum  Cp  criterion  has  also  received 
praise  for  minimizing  mean  square  error  of  prediction,  but 
its  usefulness  is  limited  to  cases  where  the  regressors  are 
fixed.  Some  have  recommended  that  the  Min  Cp  criterion 
should  not  be  used  in  practice. 

The  results  of  this  thesis  indicate  that  the  Min  Sp 
criterion  does,  in  fact,  perform  well  when  the  circumstances 
are  "nice."  However,  when  circumstance  are  poor,  the  Min  Cp 
criterion  does  better  than  Sp.  Ironically,  there  is  no  way 
to  be  sure  whether  the  conditions  the  variable  selection 
techniques  must  perform  under  are  best-case  or  worst-case, 
yet  it  seems  like  such  information  is  necessary  if  one 
intends  to  minimize  MSEP. 

Recommendations  for  Further  Research 

A  study  of  this  intent  lends  itself  to  many  follow-on 
studies.  The  methodology  and  groundwork  are  established  but 
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embellishments  will  be  necessary.  Using  more  complex  models 
would  be  one  example.  The  model  used  in  this  study  was 
simple:  four  regressors  created  the  data  with  one  eliminated 
from  further  consideration. 

A  full-blown  simulation  could  be  implemented  to  construct 
guidelines  for  usage  of  these  criteria.  For  instance  in 
this  study,  results  indicate  that  if  the  circumstances  are 
favorable,  use  the  Min  criterion.  However,  if  the 
circumstances  are  unfavorable,  use  the  Min  Cp  criterion.  By 
expanding  the  simulation  to  randomly  generate  many  true 
variables  and  many  extraneous  variables  and  then  eliminate  a 
random  number  of  variables,  guidelines  could  be  established 
for  which  technique  should  be  used  under  various  conditions 
that  seem  most  likely.  That  is,  if  it  is  likely  to  be  a 
situation  where  there  are  many  extraneous  variables  and  many 
variables  omitted,  use  one  technique,  otherwise  use  another. 
Even  though  it  is  impossible  to  determine  whether  there  are 
going  to  be  many  extraneous  variables,  one  can  assume  if  the 
dependent  variable  is  something  entirely  new,  like  a  flight 
characteristic  of  the  Soviet's  new  Blackjack  bomber, 
information  on  significant  variables  will  be  unobtainable. 
Likewise,  some  data  collected  probably  will  have  little  to 
do  with  the  dependent  variable.  In  such  circumstances  it 
would  be  useful  to  have  a  guideline  for  which  variable 
selection  criterion  should  be  implemented.  To  completely 
examine  the  usefulness  of  each  criterion  and  establish 
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guidelines,  further  research  should  include  interaction 


#  terms  and  indicator  variables. 

One  area  which  leads  to  further  research  deals  with 
Response  Surface.  The  data  range  used  for  this  thesis  was 

#  limited.  It  may  be  useful  to  expanding  the  Response  Surface 
region  to  include  negative  correlation  and  larger  sample 
sizes.  If  an  indicator  variable  was  used  to  contrast  the 

9  effect  of  dropping  a  variable,  the  loop  would  be  complete. 

That  is,  by  including  a  variable  to  keep  track  of  the 
difference  between  the  full  model  and  a  model  where  a 

#  variable  is  dropped,  one  could  quantify  the  effects  of 
failing  to  collect  data  on  all  the  significant  variables. 

In  this  thesis,  only  information  from  dropping  a  variable 

0  was  collected.  It  was  assumed  that  if  all  variables  were 

present,  the  techniques  would  perform  better;  plus  similar 
simulation  studies  recorded  results  without  dropping 
0  variables.  However,  it  would  be  worthwhile  to  quantify  the 

effects  not  including  all  significant  variables  into  the 
variable  pool . 

%  This  research  has  skimmed  the  surface  of  many  myths 

associated  with  variable  selection  techniques,  and  in 
particular  the  usefulness  of  the  Cp  criterion.  Some  authors 
^  regard  the  Min  Cp  criterion  as  secondary  to  the  Min  Sp 

criterion.  However,  the  results  of  this  simulation  do  not 
support  such  ranking.  The  two  criteria  perform  differently 
0  under  certain  circumstances.  Under  the  best-case  scenario. 
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the  Min  Sp  criterion  significantly  outperform  the  Min  Cp 
criterion.  However,  under  the  worst-case  scenario,  the  Min 
Cp  criterion  significantly  outperforms  the  Sp  criterion. 

Other  simulations  deal  with  the  number  of  correct 
variables  chosen  of  those  available  in  absolute  terms.  No 
provisions  are  made  for  circumstances  in  which  significant 
regressors  are  not  included  in  the  variable  pool. 

Therefore,  techniques  praised  as  good  variable  section 
techniques  may  not  be  as  appealing  as  originally  thought. 
This  research  indicates  this  is  the  case  with  the  Min  S„ 

P 

criterion . 
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Appendix  A:  Bar  Graphs  For  Worst-Case  Comparison 


/.  -  Case  Hesu  '  ts  Vs  '  ng  MSE 


H 

I 

2^  -t 


Figure  5.  Bar  Graph  for  results  from  MSE  under  a  Worst- 
Case  Scenario 
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Appendix  B:  Macros  to  Generate  Correlated  Data 
Using  S  on  the  ASC  or  Success  on  the  CSC 

########################################################### 

This  macro  is  designed  to  calculate  correlated  data  from 
a  random  sample.  This  macro  is  for  the  1  extraneous 
variable  case.  There  is  also  a  macro  found  for  the  3 
extraneous  variable  case. 

To  implement  this  program,  the  user  must  enter  the  S 
computer  package  by  typing  S  on  the  ASC  or  Success  on  the 
CSC. 

Once  in  S,  the  following  inputs  must  be  made: 

1.  The  4x4  correlation  matrix,  K 

>  K_na t r ix ( read { ) , 4 , 4 , by row=T ) 

>  "now  input  the  rows  of  the  matrix" 

2.  The  standard  deviation  of  the  extraneous  variables 

>  exerrstd_" value" 

3.  The  standard  deviation  of  the  independent  variables 

>  xerrstd_" value" 

4.  The  standard  deviation  of  the  €  term 

>  ye r r s td_" va 1 ue " 

5.  The  sample  size  (this  is  a  two  step  procedure) 

>  ptsperset_" sample  size" 

>  samplesize_" total  number  of  points  desired 
samplesize*number  of  runs 

The  macros  can  be  implemented  by  the  following  commands 
>source  "make. 1  error"  for  1  ex  vars  case 

>source  "make . 3error "  for  3  ex  vars  case 

The  files  which  are  implemented  by  these  macros  are  the 
following : 

1 .  make . Terror 

2 .  make . 3error 

3.  batch 

4 .  four 

5.  matgen 

6.  qmake 

The  data  that  is  generated  has  an  appended  column  of 
integers  which  is  used  in  the  SAS  runs. 
############################################################ 
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############################################################ 

The  following  must  be  typed  to  write  the  data  to  a  file. 

>  write ( t ( Z )," filename" , #) 

filename  must  be  in  quotes  and  #  is  the  number  of 
columns  of  the  data  set.  In  the  lex  case  it  is  7 
in  the  3ex  case  it  is  9- 

############################################################ 

>  #  u  an  V  are  random  noise,  mean  0,  and  std  exerrstd 

>  v_rnorm( samplesize , 0 , exerrstd) 

>  u_rnorm ( samples ize , 0 , exerrstd ) 

>  #  x1..x4  are  the  independent  variables 

>  x1 _rnorm ( samples i ze , 0 , xerrstd ) 

>  x2_rnorm ( samples ize , 0 , xerrstd ) 

>  x3_rnorm ( samples ize , 0 , xerrstd ) 

>  x4_rnorm ( samplesize , 0 , xerrstd) 

> 

>  X_cbind ( x1 , x2 , x3 , x4 ) 

>  e_eigen(K) 

>  lamb__diag  {  e$values  ) 

>  P_e$vectors 

>  XTX_t(X)%*X 

>  qq_e$values 

>  s ig_diag ( XTX ) 

>  qi.qq/sig 

>  Ql_diag< sqrt { q1  )  ) 

>  Q_Q1%*t(P) 

>  Z_X%*Q 

>  op_rnorm ( samples ize , 0 , yerrstd  ) 

>  y_Z [ , 1 ] +Z [ , 2 ] +Z [ , 3 ] +Z [ , 4 ] +op 

>  error_rnorm(samplesize,0, exerrstd ) 

>  Z_cbind ( y , Z , error ) 

> 

>#  The  following  statements  create  the  appended  integers 
> 

>  g_diag ( diag ( ptsperset ) 

>  gg_c (g, g*2 , g*3 , g*4 , g*5, g*6, g»7 , g*8 , g*9, g*l 0 ) 

>  gg_c(gg,gg+10,gg+20,gg+30,gg+40,gg+50) 

>  Z_cbind ( gg , Z , V , u ) 
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option  linesize=80; 
filename  new  'msep.dat'; 
data  new; 
infile  new; 

input  set  y  x1  x2  x3  x4  el ; 

proc  rsquare  data=new  mse  sp  cp  b; 

by  set; 

model  y=  x1  x2  x3  el  ; 
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Appendix  D  :Fortran  Program  to  Find 
the  Min  MSB,  Min  C,  and  Min  Values  and  Models 


This  program  is  designed  to  take  a  modified  SAS  listing 
from  Appendix  C  (the  modification  is  simply  deleting 
all  lines  except  for  the  raw  data  i.e.  the  number  of 
variables,  the  R^,  the  MSE,  the  Sp,  and  Cp  values,  as 
well  as  the  variables  printed.  The  program  finds  the 
correct  model  using  the  Min  MSE,  Min  Sp,  and  Min  Cp 
criteria.  In  each  case,  the  model  chosen  by  each 
technique  is  printed.  The  values  for  each  criteria  are 
also  printed. 

Statistics  are  collected  on  the  average  number  of 
variables  chosen  and  the  number  of  extraneous  variables 
chosen 

variables  num  is  the  number  of  variables  in  the  model 

this  is  the  first  column  of  the  SAS 
listing 

i,j,k  are  counters  for  do  loops. 

ptrmse , ptrsp , ptrcp  are  variables  used  to  keep 
track  of  the  min  values. 

numcp , numsp , nummse  are  the  average  number 
of  variables  chosen  by  their 
respective  technique. 


cper r , sperr , mseeer  are  the  average  number 
of  extraneous  variables  chosen 
by  their  respective  technique. 


integer  num(63) ,i,j,k, ptrmse, ptrsp 
integer  ptrcp, varsmse,varssp,varscp 
integer  check(6) 
integer  n , emse , esp, ecp 
integer  ccp,cmse,csp 

integer  chartmse(0; 3,0:3) ,chartcp{0:3,0:3) 

integer  char tsp ( 0 : 3 , 0 : 3 ) 

real  MSE(63),Sp(63),cp(63),r2(63) 

real  minmse,minsp,mincp,  numtr.se  , numcp, numsp 
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real  mseeer , cpeer , speer 

character*2  m(6,63) 

check ( 1 ) =6 

check ( 2 ) =2 1 

check ( 3 ) =4 1 

check ( 4 ) =  56 

check ( 5 ) =62 

check ( 6 ) =6  3 

do  7  i  =  0 , 3 
do  3  k  =  0 , 3 

chartmse ( i , k) =0 
cha  r tcp ( i , k ) =0 
charts p  (  i  ,  k. )  =0 
continue 
continue 


var smse=0 
varssp  =0 
varscp  =0 

#  cumemse=0 

cumesp=0 

cumecp=0 

open  (unit=10,file=" temp" ,status='old' ) 
open  (unit=11,file=' OUT ' , status= ' new ' ) 

#  do  20  k=1 , 60 

do  10  i=1 , 63 

emse=0 
esp=0 
ecp  =  0 
read (10,*) 

#  num(i),r2(i),MSE(i),Sp(i),cp(i) 

+  , ( m ( j , i ) , 3 = 1 , num ( i ) ) 

minmse= 10000 
minsp  =10000 
mincp  =1 0000 
ptrmse=0 

#  ptrcp  =0 

ptrsp  =0 

do  30  3  =  1,6 

i f ( mse ( check ( 3 ) ) . It . minmse )  then 
minmse=mse(check(3) ) 
ptrmse=check(3 ) 

#  endif 

if ( sp ( check ( 3 ) ) . 1 t . minsp )  then 
minsp=sp(check(3 ) ) 
ptrsp=check ( j ) 

endif 

i f ( cp ( check ( j )). It . mincp )  then 

#  mincp=cp ( check ( j ) ) 

ptrcp=check ( 3 ) 
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endi  f 


continue 

continue 

varsmse  =  varsmse+nu!i)  (  ptrmse ) 
varssp  =varssp  +num(ptrsp) 
varscp  =varscp  +num(ptrcp) 

do  70  n= 1 , num ( pt rmse ) 

i £ ( m ( n , pt rmse ) . EQ .  *  El  ‘  )  then 
emse=emse+1 

el sei f ( m ' n , pt rmse ) . eq . ' E2 ' )  then 
emse=emse+ 1 

el sei £{ m ( n , ptrmse ). eq E3 ' )  then 

emse=emse+1 

else 

continue 
endi  f 

continue 

do  80  n=1 , num ( ptrsp ) 

i f ( m { n , ptr sp ) . eq . ' El ' )  then 
esp=esp+1 

el sei f ( m ( n , pt r sp ) . eq . ' E2 ' )  then 
esp=esp+1 

el sei f ( m { n , pt r sp ) . eq . ' E3 ' )  then 

esp=esp+1 

else 

continue 
endi  f 
continue 

do  90  n= 1 , num ( pt rep ) 

if ( m ( n , pt rep ) . eq . ' El ' )  then 
ecp=ecp+1 

el sei f ( m ( n , pt rep ) . eq . ' E2 ' )  then 
ecp=ecp+ 1 

el sei f ( m { n , pt rep ) . eq . ' E3 ' )  then 

ecp=ecp+1 

else 

continue 
endi  f 
continue 

cumemse=cumemse+emse 

cumesp=cumesp+esp 

cumecp=cumecp+ecp 

cmse=num (ptrmse) -emse 

ccp=num(ptrcp)-ecp 

csp=num(ptrsp)-esp 

chartmse(cmse,emse)®chartmse{ emse , emse ) + 1 
chartcp(ccp,ecp)=chartcp(ccp,ecp)+l 


chartsp(csp,esp)=chartsp(csp,esp)+1 


write(11,*)  'MSE', num (ptrmse) ,mse(ptrmse) 

+  , Sp ( ptrmse ), cp ( ptrmse ) 

+  ,( m ( j , ptrmse ), j = 1 , num ( ptrmse ) ) 

write(11 ,*)  'Sp' ,num(ptrsp) ,mse(ptrsp) 

+  , Sp ( ptrsp ) , cp ( ptrsp ) , 

+  ( m ( j , ptrsp ), j =1 , num ( ptrsp ) ) 

write(11,*)  'Cp', num (ptrcp) ,mse(ptrcp) 

+  , Sp(ptrcp) , cp{ptrcp) , 

+  ( m ( j , pt rep ), j =1 , num ( ptrcp ) ) 

write (11,*)  '  ' 

wr ite ( 1 1  , *  )  '  • 

20  continue 

nummse  =  real ( varsmse ) /60 . 0 
numsp  =  real ( varssp) /60 . 0 
numcp  =  rea 1 ( va r sep ) /60 . 0 
mseeer=  real ( cumemse ) / 60 . 0 
epeer  =  real{cumecp)  /60.0 
speer  =  real(cumesp)  /60.0 
write(11,*)  'The  avg  number  of  vars  using  MSE 
was  ' ,  nummse 

write(11,*)  'The  avg  number  of  errors  from  MSE 
was ' ,  roseeer 
write ( 1 1 , * )  '  ' 

write(11,*)  'The  avg  number  of  vars  using  Sp  was', 
numsp 

write(11,*)  'The  avg  number  of  errors  from  Sp  was 
' ,  speer 

wri te ( 1 1 , * )  '  ' 

write(11,*)  'The  avg  number  of  vars  using  Cp  was 
' ,  numcp 

write(11,*)  'The  avg  number  of  errors  from  Cp  was 
'  ,  epeer 

wr ite ( 1 1 , * )  '  ' 

wri te { 1 1 , *  )  '  ' 

write(11,*)  ‘  MSE  TABLE' 

write (11,*)  '  ' 

do  100  i=0,3 

write( 11 ,*)  (chartmse(i,j) ,j=0,3) 

100  continue 

write (11,*)  '  ’ 

wri te ( 1 1 , *  )  '  ' 

write ( 11,*)  ' Sp  TABLE ' 

wr ite ( 1 1 , *  ) 

do  110  i=0,3 

write(11,*)  { chartsp ( i , j ) , j =0 , 3  ) 
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1  1  0 


continue 


1  20 


write( 11,*)*' 
writedi,*)  '  * 

write( 11,*)  'Cp  TABLE’ 
do  120  i=0,3 

write(11,*)  ( chartcp { i , j )  , j =0 , 3  ) 

continue 

END 
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Appendix  E:  Sample  Output  for  Phase  One  of  Analysis 

The  following  is  a  sample  of  the  output  for  one  run  (with  60 
replications)  for  the  Response  Surface  Methodology  Phase 
(program  found  in  Appendix  D).  The  first  column  contains 
the  technique  used.  The  second  column  is  the  number  of 
variables  chosen.  The  third,  fourth,  and  fifth  columns  are 
the  values  for  Min  MSE,  Min  Cp,  and  Min  Sp  respectively. 

The  last  column  is  the  actual  model  chosen  by  the  technique. 


MSE 

Sp 

Cp 

«r  *  *  * 

1 

1 

1 

AAA 

0.100654  1. 
0.100654  1. 
0.100654  1. 

43791e-02  -0. 
43791e-02  -0. 
43791e-02  -0. 

AAAAAAAAAAAAA 

266093  XI 
266093  XI 
266093  XI 

*********** 

A  A  A  A 

AAA 

MSE 

2 

5. 32424e-02 

8.87374e-03 

1  .  35403 

X2 

El 

Sp 

1 

6.00226e-02 

8.57466e-03 

0.898098 

El 

Cp 

*  *  *  * 

1 

*  *  * 

6. 00226e-02 

8.57466e-03 

0.898098 

El 

A  A  A  A 

*  *  * 

MSE 

3 

7 . 63447e-02 

1 . 52689e-02 

3.17057 

XI 

X2 

Sp 

1 

7.64603e-02 

1 . 09229e-02 

0.904536 

X2 

Cp 

A  A  *  A 

1 

*  *  * 

7 . 64603e-02 

1 . 09229e-02 

0.904536 

X2 

*  *  *  * 

*  *  * 

MSE 

1 

2.95358e-02 

4.21940e-03 

-0.285088 

X3 

Sp 

1 

2. 95358e-02 

4.21940e-03 

-0.285088 

X3 

Cp 

*  *  *  * 

1 

*  *  * 

2. 95358e-02 

AAAAAAAAAAAAAA 

4 . 21 940e-03 

AAAAAAAAAAAAA 

-0.285088 

AAAAAAAAAAA 

X3 

*  *  *  * 

*  *  * 

MSE 

2 

4. 95487e-02 

8.2581 2e-03 

1.35192 

X2 

X3 

Sp 

2 

4.95487e-02 

8. 2581 2e-03 

1 . 35192 

X2 

X3 

Cp 

**** 

1 

*** 

5.93601e-02 

8. 48001 e-03 

************* 

1 . 32763 

*********** 

X2 

*  *  *  * 

*  *  * 

MSE 

2 

6. 12631e-02 

1 .02105e-02 

1 . 09398 

X2 

El 

Sp 

1 

6. 46866e-02 

9.24094e-03 

0. 1 47028 

X2 

Cp 

1 

6.46866e-02 

9. 24094e-03 

0.1 47028 

X2 

El 


MSE  2 
Sp  1 
Cp  1 


MSE  1 
Sp  1 
Cp  1 


MSE  1 
Sp  1 
Cp  1 

*  ‘ 


MSE  1 

Sp  1 

Cp  1 


3.191 34e-02 

5. 31 889e-03 

1.45195 

X2 

3 . 24903e-02 

4.64148e-03 

0. 343450 

X2 

3. 24903e-02 

4.641 48e-03 

0. 343450 

X2 

6. 24057e-02 

8.91 51 Oe-03 

-0. 29991 2 

X3 

6. 24057e-02 

8.91 510e-03 

-0.299912 

X3 

6. 24057e-02 

8.91510e-03 

-0.29991 2 

X3 

3.94331e-02 

k:fc*********A*4 

5.63330e-03 

-0.741483 

*  *  * 

X2 

3.94331e-02 

5.63330e-03 

-0.741483 

X2 

3.94331e-02 

5.63330e-03 

-0.741 483 

X2 

.*************, 

2. 251 72e-02 

3 . 21 674e-03 

1 . 02995 

El 

2. 251 72e-02 

3 . 21  674e-03 

1 .02995 

El 

2 . 251 72e-02 

3. 21  674e-03 

1 .02995 

El 

MSE  1 
Sp  1 
Cp  1 

****** 

MSE  1 


3.95394e-02 
3. 95394e-02 
3.95394e-02 


2 . 43904e-02 

2 . 54640e-02 

2 . 54640e-02 

4.065076-03 

3.637726-03 

3.637726-03 

1 .66245 
0.756217 
0.756217 

X2 

X2 

X2 

X3 

3 . 30050e-02 

5.500836-03 

1  .  2961  1 

XI 

X2 

3. 30050e-02 

5.500836-03 

1 . 2961 1 

XI 

X2 

3. 30050e-02 

5.500836-03 

1 . 2961 1 

XI 

X2 

.***************** 

3. 95879e-02 

6.597996-03 

1 . 55024 

XI 

X2 

3. 95879£-02 

6.597996-03 

1 . 55024 

XI 

X2 

3 . 958  79e-02 

6.597996-03 

1 . 55024 

XI 

X2 

************,1 

^************4 

r***************** 

8.972546-02 

1 .281796-02 

-0.889888 

XI 

8.972546-02 

1 .281796-02 

-0.889888 

XI 

8.972546-02 

1 .281796-02 

-0.889888 

XI 

5.64849e-03 

5.64849e-03 

5.64849e-03 


c**************** 

•0.658592  X2 
■0.658592  X2 
■0.658592  X2 


79 


MSE 

2 

3. 60757e-02 

6.01 262e-03 

1 .61985 

X2 

El 

Sp 

1 

3. 66450e-02 

5. 23500e-03 

0.524038 

El 

Cp 

h-kffk 

1 

*** 

3. 66450e-02 

5.23500e-03 

0.524038 

AAAAAAAAAAA 

El 

*  *  *  * 

*  *  * 

MSE 

2 

5. 85844e-02 

9.76407e-03 

2.28246 

XI 

X2 

Sp 

2 

5 . 85844e-02 

9.76407e-03 

2 .28246 

XI 

X2 

Cp 

A  *  * 

2 

*  *  * 

5. 85844e-02 

9-76407e-03 

2.28246 

XI 

A  A  A  A 

X2 

AAA 

MSE 

2 

4.41 387e-02 

7 . 35645e-03 

1 . 80789 

X2 

X3 

Sp 

1 

4 . 42947e-02 

6. 32782e-03 

0.661062 

X2 

Cp 

1 

4 . 42947e-02 

6. 32782e-03 

0.661062 

X2 

**** 

AAA 

************** 

************* 

AAAAAAAAAAA 

*  *  *  * 

AAA 

MSE 

1 

0.100894  1. 

44134e-02  -0. 

614306  X3 

Sp 

1 

0.100894  1. 

44134e-02  -0. 

614306  X3 

Cp 

it  it  *  it 

1 

AAA 

0.100894  1. 

44134e-02  -0. 

614306  X3 

A  A  A  A 

AAA 

MSE 

1 

3. 93769e-02 

5.62527e-03 

-0.7901 19 

X3 

Sp 

1 

3. 93769e-02 

5.62527e-03 

-0.7901 1 9 

X3 

Cp 

A  A  A  A 

1 

*  *  * 

3.93769e-02 

5.62527e-03 

-0.790119 

AAAAAAAAAAA 

X3 

*  *  *  * 

AAA 

MSE 

1 

4. 66999e-02 

6.671 42e-03 

-0. 101598 

El 

Sp 

1 

4.66999e-02 

6.67142e-03 

-0.101 598 

El 

Cp 

A  A  A  A 

1 

*  *  * 

4 . 66999e-02 

6.671 42e-03 

-0.101598 

El 

*  *  *  * 

AAA 

MSE 

1 

4 . 52970e-02 

6.47101e-03 

-0.946171 

X2 

Sp 

1 

4 . 52970e-02 

6.47l01e-03 

-0.946171 

X2 

Cp 

1 

4.52970e-02 

6.47101e-03 

-0.946171 

X2 

MSE 

3 

2. 84832e-02 

5.69664e-03 

3 . 30669 

XI  X2 

Sp 

1 

3 . 08908e-02 

4 . 41 297e-03 

1 . 67367 

X2 

Cp 

*  *  *  A 

1 

A  *  A  1 

3.08908e-02 

4 . 41 297e-03 

1 .67367 

kAAAAAAAAAA 

X2 

A  A  A  A  A  A  A 

MSE 

1 

4. 47498e-02 

6. 39283e-03 

-0.492574 

El 

Sp 

1 

4 . 47498e-02 

6. 39283e-03 

-0.492574 

El 

Cp 

1 

4. 47498e-02 

6. 39283e-03 

-0.492574 

El 

X3 
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MSE 

1 

6. 1 0468e-02 

8.72097e-03 

-0.754845 

X2 

• 

Sp 

1 

6. 1 0468e-02 

8. 72097e-03 

-0.754845 

X2 

Cp 

1 

6. 1 0468e-02 

8.72097e-03 

-0.754845 

X2 

MSE  1  2.40736e-02 
Sp  1  2.40736e-02 
Cp  1  2.40736e-02 


3 . 43908e-03 
3 . 43908e-03 
3. 43908e-03 


1.06484  X3 
1.06484  X3 
1.06484  X3 


********1 


MSE 

1 

4 . 34820e-02 

6.211 72e-03 

0.251277 

XI 

Sp 

1 

4 . 34820e-02 

6.21 1 72e-03 

0.251277 

XI 

Cp 

1 

4 . 34820e-02 

6.21 1 72e-03 

0. 251 277 

XI 

MSE  2  2.60387e-02 
Sp  2  2.60387e-02 
Cp  2  2.50387e-02 


***********1 


MSE  3  1.14088e-02 
Sp  3  1.14088e-02 
Cp  3  1.14088e-02 


4 . 33979e-03 
4. 33979e-03 
4. 33979e-03 


2 . 281  77e-03 
2. 281 77e-03 
2.28177e-03 


1  .  65346 

X2 

X3 

1 .65346 

X2 

X3 

1 .65346 

X2 

A  *  A  A 

X3 

*  *  * 

3.04763 

XI 

X3 

El 

3.04763 

XI 

X3 

El 

3.04763 

XI 

X3 

El 

MSE 

4 

1 . 1 1 825e-02 

2 . 79562e-03 

5.00000 

XI 

X2 

X3 

El 

Sp 

2 

1 .66321e-02 

2 . 77202e-03 

6.41136 

XI 

El 

Cp 

4 

1.11 825e-02 

2.79562e-03 

5.00000 

XI 

X2 

X3 

El 

MSE  3  1.24700e-02 
Sp  1  1.44043e-02 
Cp  1  1.44043e-02 

MSE  1  3.17437e-02 
Sp  1  3.17437e-02 
Cp  1  3.17437e-02 


2 . 49400e-03 
2.05776e-03 
2.05776e-03 


3.16464  X2  X3  El 
1.95435  X2 
1.95435  X2 


4.53481e-03  -0.727781  X3 
4.53481e-03  -0.727781  X3 
4.53481e-03  -0.727781  X3 


MSE 

3 

3 . 45629e-02 

6.91257e-03 

3.52295 

XI 

Sp 

1 

4 . 07384e-02 

5.81977e-03 

2 .67969 

X3 

Cp 

1 

4.07384e-02 

5. 81 977e-03 

2.67969 

X3 
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MSE 

1 

6.70823e-02 

9 . 5831 8e-03 

-0.670246 

X3 

Sp 

1 

6. 70823e-02 

9. 5831 8e-03 

-0.670246 

X3 

Cp 

1 

6. 70823e-02 

9.58318e-03 

-0.670246 

X3 

****** 

**************** 

************* 

t********** 

*  *  *  * 

*  *  * 

MSE 

2 

3. 87667e-02 

6.461 1 1e-03 

1  .  12307 

X3 

El 

Sp 

1 

4.21 533e-02 

6.02190e-03 

0.366424 

El 

Cp 

*  *  *  * 

1 

*  * 

4.21 533e-02 

6.02190e-03 

0.366424 

t********** 

El 

*  *  *  * 

*  *  * 

MSE 

1 

4 . 98360e-02 

7.11 942e-03 

-0.172478 

X2 

Sp 

1 

4. 98360e-02 

7.11 942e-03 

-0. 172478 

X2 

Cp 

1 

4. 98360e-02 

7.11 942e-03 

-0. 172478 

X2 

****** 

**************** 

************* 

It********** 

*  *  *  * 

*  *  * 

MSE 

1 

1 . 9861 9e-02 

2.83742e-03 

-0. 1 08219 

El 

Sp 

1 

1 . 9861 9e-02 

2.83742e-03 

-0.1 08219 

El 

Cp 

*  *  «  * 

1 

*  * 

1 . 9861 9e-02 

2.83742e-03 

-0. 108219 

El 

*  *  *  * 

*  *  * 

MSE 

2 

2.27566e-02 

3.79276e-03 

1 .19987 

XI 

X2 

Sp 

2 

2. 27566e-02 

3.79276e-03 

1 . 1 9987 

XI 

X2 

Cp 

1 

2.67326e-02 

3 . 81 894e-03 

0.981005 

X2 

MSE 

1 

2.31 557e-02 

3 . 30796e-03 

-0.325469 

X2 

Sp 

1 

2 . 31  557e-02 

3 . 30796e-03 

-0.325469 

X2 

Cp 

1 

2 . 31  557e-02 

3 . 30796e-03 

-0.325469 

X2 

MSE 

2 

2. 96808e-02 

4.94681e-03 

1  .  36022 

XI 

X3 

Sp 

2 

2. 96808e-02 

4.94681e-03 

1 . 36022 

XI 

X3 

Cp 

2 

2.96808e-02 

4.94681e-  03 

1 .36022 

XI 

X3 

MSE 

3 

2.72695e-02 

5. 45390e-03 

3.00697 

XI 

X2 

Sp 

3 

2. 72695e-02 

5.45390e-03 

3.00697 

XI 

X2 

Cp 

3 

2. 72695e-02 

5.45390e-03 

3.00697 

XI 

X2 

MSE 

2 

4 . 81 207e-02 

8.02012e-03 

1 .77244 

X3  El 

Sp 

1 

5. 1 471 8e-02 

7 .3531  1 e-03 

1 .05649 

X3 

Cp 

1 

5.1 471 8e-02 

7 .3531  1e-03 

1 .05649 

X3 
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MSE  1  0.124977  1.78539e-02  -0.908955  X2 
Sp  1  0.124977  1.78539e-02  -0.908955  X2 
Cp  1  0.124977  1.78539e-02  -0.908955  X2 


MSE 

1 

3.86961e-02 

5. 52802e-03 

-0.554765 

X3 

Sp 

1 

3.86961e-02 

5.52802e-03 

-0.554765 

X3 

Cp 

1 

3 . 86961 e-02 

5. 52802e-03 

-0.554765 

X3 

MSE 

2 

1  .  07762e-02 

1 . 79603e-03 

1 .05727 

X3 

Sp 

1 

1 . 25208e-02 

1 . 78869e-03 

0.715458 

El 

Cp 

1 

1 . 25208e-02 

1 .78869e-03 

0.71 5458 

El 

MSE 

4 

2.31  882e-02 

5. 79705e-03 

5.00000 

XI 

X2 

Sp 

2 

2. 46525e-02 

4.1 0876e-03 

3.44206 

XI 

El 

Cp 

2 

2 . 46525e-02 

4.  10876e-03 

3 .44206 

XI 

El 

MSE 

1 

6.041 22e-02 

8.63031e-03 

-0.792417 

XI 

Sp 

1 

6. 041 22e-02 

8.63031e-03 

-0.792417 

XI 

Cp 

*  *  *  * 

1 

*  * 

6. 041 22e-02 

8.63031e-03 

-0.792417 

XI 

*  *  * 

MSE 

1 

4. 62591 e-02 

6.60844e-03 

0.381250 

El 

Sp 

1 

4.62591e-02 

6.60844e-03 

0.381250 

El 

Cp 

1 

4 . 62591 e-02 

6.60844e-03 

0.381250 

El 

MSE  1  0.110044  1.57205e-02  -0.301020  XI 
Sp  1  0.110044  1.57205e-02  -0.301020  XI 
Cp  1  0.110044  1.57205e-02  -0.301020  XI 

MSE  1  0.116894  1.66991e-02  -0.512063  El 
Sp  1  0.116894  1.66991e-02  -0.512063  El 
Cp  1  0.116894  1.66991e-02  -0.512063  El 


MSE 

3 

5. 91 980e-02 

1  .  1  8396e-02 

3.09286 

XI 

Sp 

1 

6. 1  3972e-02 

8. 771 02e-03 

1 . 04274 

X2 

Cp 

1 

6. 1  3972e-02 

8.77102e-03 

1 .04274 

X2 

X3  El 
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MSE 

3 

1  .  67376e-02 

3 . 34753e-03 

3.37179 

X2 

X3 

El 

Sp 

3 

1  .67376e-02 

3.34753e-03 

3.37179 

X2 

X3 

El 

Cp 

3 

1  .  67376e-02 

3 . 34753e-03 

3.37179 

X2 

X3 

El 

MSE  1  8.60897e-02 
Sp  1  8.60897e-02 
Cp  1  8.60897e-02 


1 . 22985e-02 
1 . 22985e-02 
1  - 22985e-02 


0.261788 

0.261788 

0.261788 


El 

El 

El 


MSE 

3 

1  .  1  9855e-02 

2 . 39709e-03 

3.05798 

XI 

X2 

Sp 

2 

1  .  26033e-02 

2. 10055e-03 

2 .20518 

XI 

X3 

Cp 

2 

1  .  26033e-02 

2.1 0055e-03 

2.20518 

XI 

X3 

A  A  A  A 

*  *  * 

AAAAAAAAAAAAAA4 

c************* 

********** 

*  *  *  * 

*  *  * 

MSE 

2 

3 . 42009e-02 

5. 7001 5e-03 

1 . 36503 

XI 

El 

Sp 

1 

3 . 86254e-02 

5.51791e-03 

0.924681 

El 

Cp 

A  A  A  A 

1 

*  *  * 

3 . 86254e-02 

5.51791e-03 

0.924681 

El 

*  *  *  * 

*  *  * 

MSE 

1 

3 . 65205e-02 

5. 21 721 e-03 

-9. 1 8206e 

-02 

X3 

Sp 

1 

3 . 65205e-02 

5. 21 721 e-03 

-9. 1 8206e 

-02 

X3 

Cp 

A  A  A  A 

1 

*  *  * 

3 . 65205e-02 

AAAAAAAAAAAAAAif 

5.21721e-03 

-9. 1 8206e 

-02 

A  A  A  A 

X3 

*  *  * 

MSE 

2 

2 . 501 95e-02 

4  .  1 6992e-03 

1 .49327 

X2 

El 

Sp 

1 

2 . 60450e-02 

3.72071e-03 

0.535332 

El 

Cp 

A  A  A  A 

1 

*** 

2.60450e-02 

3 .7207 1 e-03 

0.535332 

El 

*  *  *  * 

AAA 

MSE 

1 

4. 73777e-02 

6. 76825e-03 

-0.485272 

El 

Sp 

1 

4 . 73777e-02 

6.76825e-03 

-0.485272 

El 

Cp 

A  A  A  A 

1 

AAA 

4. 73777e-02 

6.76825e-03 

-0.485272 

El 

*  *  *  * 

AAA 

MSE 

2 

4.89740e-02 

8. 16234e-03 

1 .74329 

XI 

X3 

Sp 

1 

5. 48795e-02 

7.83993e-03 

1 . 35524 

XI 

Cp 

A  A  A  A 

1 

*  *  * 

5. 48795e-02 

AAAAAAAAAAAAAAl 

7.83993e-03 

1 . 35524 

XI 

*  *  *  * 

AAA 

MSE 

1 

2 . 25924e-02 

3.22748e-03 

0.72994? 

El 

Sp 

1 

2 . 25924e-02 

3. 22748e-03 

0.729942 

El 

Cp 

1 

2. 25924e-02 

3.22748e-03 

0.729942 

El 

X3 


The  avg  number  of  vars  using  MSE  was 
The  avg  number  of  errors  from  MSE  was 


1 .71667 
0-416667 


The  avg  number  of  vars  using  Sp  was 
The  avg  number  of  errors  from  Sp  was 

The  avg  number  of  vars  using  Cp  was 
The  avg  number  of  errors  from  Cp  was 

where 

errors  stands  for  extraneous  terms 
vars  stands  for  the  number  of  variables  chosen. 

To  find  the  response  in  the  Response  Surface  Phase,  the 
following  equation  was  used. 

(1-avg  errors)/(avg  vars) 

For  example,  the  following  is  the  calculation  for  the  C,. 
case . 

( 1 -. 33333 ) /I . 2667  =  .5263 

This  value  was  used  as  one  of  the  64  Cp  response  values  in 
the  response  surface  phase. 


1 . 26667 
0. 333333 

1 . 26667 
0.333333 


e 


Appendix  F:  Fortran  Program  for  finding 
TMSEP  values,  rank  ordering  them  and 
Comparing  the  three  criteria 


* 

*  This  program  is  designed  to  take  a  modified  SAS  program 

*  and  an  existing  data  set  and  find  the  TMSEP  the  models. 

*  The  modifications  necessary  are  the  same  as  discussed  in 

*  Appendix  D. 

* 

*  The  one  exception  is  that  the  SAS  listing  prints  out  the 

*  coefficient  values.  When  the  variable  is  not  included  in 

*  the  model  a  is  left.  This  symbol  must  be  replaced 

*  with  a  "0". 

*  To  accomplish  this  the  following  UNIX  editing  command  can 

*  be  used. 

* 

*  g/  \.  /s//  0  /g 

if 

*  This  command  will  make  the  appropriate  substitution. 

it 

*  In  this  program  all  integer  values  are  counters  (for  do 

*  loops  except  for  the  arrays. 

* 


* 

* 

it 

* 

it 

* 

A 

* 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 

A 


variable  ptrmse , ptrsp, ptrcp  are  used  to  keep  track 

of  the  lowest  value  found  using 
MSE,  Sp,  and  Cp  respectively. 

spcount , equa Icount , spcpct , cpcount  are  all 
counters  for  keeping  track  of  the 
number  of  times  the  lowest  TMSEP  value 
was  found  by  Min  Sp,  all  techniques, 
Min  Sp  and  Min  Cp,  and  Min  Cp  only. 

r2,mse,sp,cp  are  the  arrays  for  r^,  mse,  Sp 
and  C^. 

K 

bO  and  betas  are  the  values  for  the 
coef  f icients 

y s sepmse , yssepsp , ys sepcp  are  the  Sum-of 

-Squares  for  the  respective  technique. 


*  ymsepmse , ymsepsp , ymsepcp  are  the  MSEP 

*  values  using  the  respective  technique. 

* 

* 


integer  i,k,j ,p,h,num( 15) , ptrmse, ptrsp, ptrcp 
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integer  spcount.equalcnt, set,check(4) ,spcpct,cpcount 

real  b0l15),r2(15),mse(15),sp(15) 

real  betas(4, 15) ,minmse,mincp,minsp 

real  x ( 4 ) , ex , x3ex3 ( 4 ) , bOmse 

real  bOsp , bOcp , ypredmse , ypredsp 

real  ypredcp, ymsepmse , ymsepsp , ymsepcp 

real  ys sepmse , y s s epsp , y s sepcp , y 

real  cp ( 1 5 ) 

spcpct=0 
equa lcnt=0 
spcount=0 
cpcount=0 
msecount=0 
check ( 1 ) =  4 
check ( 2 ) = 1 0 
check ( 3 ) =1 4 
check ( 4 ) = 1 5 

open(unit=12, file=  'msepi .lis ' , status=’old' ) 
open (unit=1 0, file= ' msepi .dat' ,status='old' ) 
open(unit=11 ,file=’msep1-out* ,status='new' ) 

do  20  k=1  ,  60 
do  10  i  =  1  ,  1  5 

read  (12,*)  num(i),  r2(i),  mse(i),  sp(i),  cp(i), 

bOli) 

+  ,betas(1 ,i) ,betas(2,i) ,betas(3,i) ,betas(4,i) 
minmse  =  10000 
minsp  =  10000 
mincp  =  10000 
yssepmse=0 
yssepsp  =0 
yssepcp  =0 
ptrmse  =  0 
ptrsp  =  0 
ptrcp  =  0 
do  30  j  =  1  , 4 

i f ( mse ( check ( j )). It . minmse )  then 
minmse  =  mse ( check ( j ) ) 
ptrmse  =  check(j) 
endif 

if  ( sp ( check ( j )). 1 t . minsp)  then 
minsp  *  sp(check(j)) 
ptrsp  *  check( j ) 
endif 

if  { cp ( check ( j )). 1 t . mincp )  then 
mincp  =  cp(check{j)) 
ptrcp  *  check ( j ) 
endif 

30  continue 

1 0  continue 
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do  50  h=  1 , 20 

read  (10,*)  set , y , x ( 1 ) , x ( 2 ) , x ( 3 ) , x ( 4 ) , ex 

ypredmse=  0 

ypredsp  =  0 

ypredcp  =  0 

yactual  =  0 

bOmse  =  bO(ptrmse) 

bOsp  =  bO(ptrsp) 

bOcp  =  bO(ptrcp) 

ypredmse=  bOmse 

ypredsp  =  bOsp 

ypredcp  =  bOcp 

x3ex3(1)=  x(1) 

x3ex3(2)=  x(2) 

x3ex3(3)=  x(3) 

x3ex3 ( 4 ) =  ex 

do  60  p= 1 , 4 

yactual  =  yactual+x(p) 

60  continue 

do  7  0  p= 1  , 4 

ypredmse  =  ypredmse  +  beta s ( p , p t rmse ) *x3ex3 (  p ) 
ypredsp  =  ypredsp  +  betas ( p , ptrsp )  *x3ex3(p) 
ypredcp  =  ypredcp  +  betas ( p , ptrcp )  *x3ex3(p) 

70  continue 

yssepmse  *  ( ( ypredmse-yactual ) **real ( 2 ) )  +  yssepmse 

yssepsp  =  ((ypredsp  -yactual ) **real ( 2 ) )  +  yssepsp 
yssepcp  =  ((ypredcp  -yactual ) **real ( 2 ) )  +  yssepcp 
50  continue 

ymsepmse  =  yssepmse  /  ( 20-num ( ptrmse ) ) 
ymsepsp  =  yssepsp  /  ( 20-num ( ptrsp ) ) 
ymsepcp  =  yssepcp  /  ( 20-num ( ptrcp ) ) 

i f ( ymsepsp . 1 t . ymsepcp )  spcount=spcount+1 
i f ( ymsepsp . eq . ymsepmse )  equa lcnt=equa lcnt+ 1 
if ( ymsepcp. It. ymsepsp)  cpcount=cpcount+ 1 
if (ymsepmse. It. ymsepsp)  msecount=msecount+1 
if( ym sepsp.eq. ymsepcp. and. ymsepmse.gt. ymsepsp) 
spcpcnt=spcpct+1 

write (11,*)  ' run  '  ,  k 

write (11,*)  '  ' 

write ( 1 1 , * )  '  ' 

write(11,*)  '  Sp  MSEP  =  ymsepsp 

write(11,*)  '  Cp  MSEP  »  ',  ymsepcp 

write(11,*)  '  MSE  MSEP  =  ',  ymsepmse 

write (11,*)  '  ' 

wr i te ( 1 1 , * )  '  ' 

20  continue 

write ( 1 1 , *  )  '  ' 

write (11,*)  '  ' 

write (11,*)  '  ' 
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wr ite { 1 1 , * ) 

write ( 1 1 , * ) 

write (11,*) 

write ( 1 1 , * ) 

wr ite ( 1 1 , * ) 
END 


'  Number  of  times  Sp  had  smallest  MSEP 
' , spcount 

'  Number  of  times  Cp  had  smallest  MSEP 
' , cpcount 

'  Number  of  times  MSE  had  smallest  MSEP 
' , msecount 

'  Number  of  times  Cp  and  Sp  were  equal 
' , spcpct 

'  All  MSEPs  were  equal equalcnt 


The  following  is  a  sample  output  for  phase  two  of  the 


analysis.  This  output  is  from  the  program  found  in  Appendix 
F.  The  TMSEP  values  from  each  method  is  given,  then  the 
rank  ordered  TMSEP  values  for  each  subset  of  values  is 
given . 


run 

1 

Sp 

TMSEP  = 

8. 1 625051 E-04 

Cp 

TMSEP 

8. 1 625051 E-04 

MSE 

TMSEP  = 

8 . 1 625051 E-04 

TMSEPs  FOR  RUN  1 


1 

5.387531 1 E-04 

2 

6.2458171E-04 

3 

6. 281 6928E-04 

4 

7 . 8544696E-04 

5 

8. 1625051 E-04 

6 

8.6704991 E-04 

7 

8.7211 549E-04 

8 

8. 974701 OE-04 

9 

9. 1 31 7965E-04 

1  0 

9. 565791 7E-04 

1  1 

1 . 31 82798E-03 

1  2 

1 . 4625470E-03 

1  3 

1 .7037858E-03 

1  4 

1 . 8694992E-03 

15 

9. 8459609E-03 

run 

2 

Sp 

TMSEP  * 

3. 9654067E-03 

Cp 

TMSEP  = 

3.9654067E-03 

MSE 

TMSEP  = 

3. 9654067E-03 

TMSEPs  FOR  RUN  2 
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1 

2.0111 492E-03 

2 

2. 24251  81  E-03 

3 

2. 2925893E-03 

4 

3.9654067E-03 

5 

4. 0591  503E-03 

6 

4 . 1  8531  75E-03 

7 

4. 3666265E-03 

8 

4. 4225999E-03 

9 

4.4807326E-03 

1  0 

4 . 484261  9E-03 

1  1 

5. 8024856E-03 

1  2 

5.8971639E-03 

1  3 

6. 2541  468E-03 

1  4 

6. 5273000E-03 

1  5 

1  .  0629051  E-02 

run 

3 

Sp 

TMSEP 

=  1  .  7  1  87677E-02 

Cp 

TMSEP 

=  1  .  71  87677E-02 

MSE 

TMSEP 

=  1  .  7 1  87677E-02 

TMSEPs  FOR 

RON  3 

1 

1  .0465025E-02 

2 

1 . 2726073E-02 

3 

1  .  3100259E-02 

4 

1 . 6458001  E-02 

5 

1 .7187677E-02 

6 

1 . 7703351 E-02 

7 

1 . 91 49231 E-02 

8 

2. 01 30737E-02 

9 

2.1833103E-02 

10 

2. 2608828E-02 

1  1 

2.3035588E-02 

1  2 

2. 3823561 E-02 

1  3 

2. 5661 029E-02 

1  4 

2. 7239855E-02 

1  5 

3.0690275E-02 

run 

4 

Sp 

TMSEP 

=  2.8920151E-02 

Cp 

TMSEP 

=  2. 89201 51 E-02 

MSE 

TMSEP 

=  2, 89201 51 E-02 

91 


TMSEPs  FOR  RUN 


4 


1 

1 . 898901 2E-02 

2 

1 . 9804025E-02 

3 

2.0740228E-02 

4 

2.11 99832E-02 

5 

2. 1 93451 7E-02 

6 

2. 21 68593E-02 

7 

2 . 2491 228E-02 

8 

2. 345591 6E-02 

9 

2. 3542039E-02 

1  0 

2 . 464681 3E-02 

1  1 

2 . 5090951 E-02 

1  2 

2.6086060E-02 

1  3 

2 . 6278302E-02 

1  4 

2.8023606E-02 

1  5 

2 . 89201 51 E-02 

run 

5 

Sp 

TMSEP 

2. 5275916E-02 

Cp 

TMSEP 

2. 527591 6E-02 

MSE 

TMSEP 

2.5275916E-02 

TMSEPs  FOR 

RUN  5 

1 

1  .  1  425762E-02 

2 

1  .  2566670E-02 

3 

1 . 2761  764E-02 

4 

1  .  3455779E-02 

5 

1  .  4553362E-02 

6 

1  .  547341  2E-02 

7 

1  .  5937347E-02 

8 

1 . 6735807E-02 

9 

1  .  7853793E-02 

1  0 

1 . 9062478E-02 

1  1 

2. 1006519E-02 

1  2 

2.5275916E-02 

1  3 

2.7103966E-02 

1  4 

2.7148893E-02 

1  5 

2.9038427E-02 

run 

6 

Sp  TMSEP 
Cp  TMSEP 
MSE  TMSEP 


1 . 5230293E-02 
1  . 5230293E-02 
1 . 5230293E-02 


TMSEPs  FOR  RUN 


6 


• 

2 

3 

4 

5 

6 

•  7 

8 

9 

1  0 
^  1 
1  2 

•  1  3 

1  4 
1  5 

run 


•  Sp  TMSEP 

Cp  TMSEP 
MSE  TMSEP 


1 . 1 1 89967E-02 
1 . 1 91 4390E-02 
1 . 2381 560E-02 
1 .251 3387E-02 
1 . 321 9284E-02 
1 . 34441 55E-02 
1 . 5230293E-02 
1 . 5570977E-02 
1 . 5827768E-02 
1 . 61 45628E-02 
1 . 65541 OOE-02 
1 . 6587460E-02 
1 . 6841 978E-02 
1 . 772381 7E-02 
2. 0091 623E-02 
7 


8.427957 1 E-03 
8.4279571E-03 
8. 4279571 E-03 


TMSEPs  FOR  RUN 


7 


run 


1  8.4279571E-03 

2  8.6327204E-03 

3  8.6681 442E-03 

4  8. 7661 548E-03 

5  8.7986309E-03 

6  9. 2602726E-03 

7  9. 5974831 E-03 

8  1 . 0461 051 E-02 

9  1 . 0792202E-02 

10  1 .084731 5E-02 

11  1 .0878014E-02 

12  1 . 0926300E-02 

13  1 . 1 085336E-02 

14  1 . 1 660442E-02 

15  3.0683922E-02 

8 


Sp  TMSEP 
Cp  TMSEP 
MSE  TMSEP 


1 . 91 74700E-03 
1 . 91 74700E-03 
1 . 0391 608E-02 


93 


TMSEPs  FOR  RUN 


8 


• 

1 

1.91 74  ^OOE-03 

2 

2 .  1  675059E-03 

3 

2 . 4701  47  3E-03 

4 

5. 9353570E-03 

5 

8. 1 103193E-03 

6 

1 .0391 608E-02 

• 

7 

1 . 0793708E-02 

8 

1 . 205051 9E-02 

9 

1 . 2089508E-02 

1  0 

1 . 2366496E-02 

1  1 

1 . 2528760E-02 

1  2 

1 . 2863966E-02 

• 

1  3 

1 . 6300550E-02 

1  4 

1 . 76291 99E-02 

1  5 

1 . 9998444E-02 

run 

9 

Sp 

TMSEP  = 

2.9792758E-03 

• 

Cp 

TMSEP  = 

2.9792758E-03 

MSE 

TMSEP  = 

2.9792758E-03 

TMSEPs  FOR  RUN  9 

1 

2. 5938307E-03 

2 

2.8756007E-03 

3 

2.9792758E-03 

4 

3.0860456E-03 

5 

3.891  3097E-03 

6 

4.2111641 E-03 

7 

4.4655493E-03 

8 

4. 7994438E-03 

9 

5. 4206038E-03 

10 

7.9978053E-03 

1  1 

8.4198406E-03 

1  2 

9. 3529476E-03 

1  3 

9. 8962309E-03 

1  4 

1 . 1 325269E-02 

1  5 

1 . 5649561 E-02 

run 

10 

Sp  TMSEP 

=  3. 405821 3E-03 

Cp  TMSEP 

=  3.405821 3E-03 

MSE  TMSEP 

=  9. 9750860E-03 

TMSEPs  FOR 

RUN  1 0 

94 


1 

2. 1 047422E-03 

2 

2.531 391 6E-03 

3 

3.405821 3E-03 

4 

4.1 502793E-03 

5 

4.9941 1 30E-03 

6 

9. 2351 809E-03 

7 

9. 4529847E-03 

8 

9. 9750860E-03 

9 

1 . 0221 747E-02 

1  0 

1 . 02697 1 6E-02 

1  1 

1 . 1 1 70504E-02 

1  2 

1 . 4728523E-02 

1  3 

1 . 691 4524E-02 

1  4 

1 . 9482933E-02 

1  5 

2.2141431 E-02 

run 

1  1 

Sp 

TMSEP  = 

1  .  8264394E-02 

Cp 

TMSEP 

1 . 8264394E-02 

MSE 

TMSEP 

2 . 656391 6E-02 

TMSEPs  FOR  RUN  1  1 

1 

5.1681511 E-03 

2 

6. 1  S’  9288E-03 

3 

6. 7232223E-03 

4 

9.9291  867E-03 

5 

1  .071  1449E-02 

6 

1  .  0777702E-02 

7 

1  .  1  229556E-02 

8 

1  .  384831  4E-02 

9 

1  .  7541 861  E-02 

1  0 

1 . 8264394E-02 

1  1 

1 .8279061  E-02 

1  2 

1 . 920841  3E-02 

1  3 

2 . 3745688E-02 

1  4 

2. 656391 6E-02 

1  5 

2. 8645089E-02 

run 

1  2 

Sp 

TMSEP  = 

6. 1  405003E-03 

Cp 

TMSEP  = 

6. 1 405003E-03 

MSE 

TMSEP 

1 . 3503841 E-02 

1  2 


TMSEPs  FOR  RUN 


1 

1 . 8639894E-03 

2 

1  .931  5871E-03 

3 

2. 0497970E-03 

4 

2 . 2980263E-03 

5 

2. 4225425E-03 

6 

2.71 20986E-03 

7 

6. 1 405003E-03 

8 

7 . 1  4281  62E-03 

9 

7 . 2628092E-03 

1  0 

8. 3331 009E-03 

1  1 

8. 9695957E-03 

1  2 

1  .  3503841  E-02 

1  3 

1  .  4081  51  4E-02 

1  4 

1  .  4754524E-02 

1  5 

1 . 5624065E-02 

run 

1  3 

Sp 

TMSEP 

=  3. 25881 69E-02 

Cp 

TMSEP 

=  3 . 25881  69E-02 

MSE 

TMSEP 

=  4 . 07401  1  OE-02 

TMSEPs  FOR  RUN 


1 

7.9008838E-04 

2 

2. 3885965E-03 

3 

4.2190668E-03 

4 

4.701 101 7 E- 03 

5 

6.7453287E-03 

6 

8. 6223893E-03 

7 

9.0424791 E-03 

8 

1  .41  71  562E-02 

9 

1 . 5555089E-02 

1  0 

1 . 6289905E-02 

1  1 

2. 1 787029E-02 

1  2 

3.2588169E-02 

1  3 

3.4396078E-02 

1  4 

4.07401 10E-02 

1  5 

4. 3201 827E-02 

run 

1  4 

Sp 

TMSEP 

=  1  .  1 8491 47E-02 

Cp 

TMSEP 

=  1 . 1 8491 47E-02 

MSE 

TMSEP 

*  1  .  1 8491  47E-02 

TMSEPs  FOR  RUN 


96 


1 

4. 8623332E-03 

2 

4, 9072048E-03 

3 

5.11 56250E-03 

4 

5. 8454480E-03 

5 

7 . 1  764030E-03 

6 

7 , 8983689E-03 

7 

8. 7281 801 E-03 

8 

1 . 1 528674E-02 

9 

1 . 1 6751 53E-02 

1  0 

1  .  1  8491  47E-02 

1  1 

1  .  2552307E-02 

1  2 

1  .  3260029E-02 

1  3 

1  .  5406845E-02 

1  4 

1 . 6281 059E-02 

1  5 

1  .  999681  3E-02 

run 

1  5 

Sp 

TMSEP 

=  7 . 47934C3E-03 

Cp 

TMSEP 

=  1  .  7  1  1  6755E-02 

MSE 

TMSEP 

=  1  .  7  1  1  6755E-02 

TMSEPs  FOR  1 

RUN  1  5 

1 

6. 61  61 933E-03 

2 

6. 8892450E-03 

3 

7 . 4793408E-03 

4 

7 , 7769053E-03 

5 

8. 791 8332E-03 

6 

9. 0924781 E-03 

7 

9. 1 773802E-03 

8 

9. 741 2989E-03 

9 

9.811 4768E-03 

1  0 

1 . 1 472788E-02 

1  1 

1 .7001675E-02 

1  2 

1 .71 16755E-02 

1  3 

1 . 7942483E-02 

1  4 

1 . 8224856E-02 

1  5 

1 . 91 86204E-02 

run 

16 

Sp 

TMSEP 

=  2 . 461 6949E-02 

cp 

TMSEP 

=  2. 461 6949E-02 

MSE 

TMSEP 

*  2 . 461 6949E-02 

TMSEPs  FOR  RUN 


1  6 


97 


► 


1 
2 

3 

4 

5 

6 

7 

8 
9 

1  0 
1  1 

>  12 

1  3 
1  4 
1  5 

run 


Sp  TMSEP 
Cp  TMSEP 
MSE  TMSEP 


TMSEPs  FOR 

1 

2 

3 

4 

5 

6 

7 

8 
9 

1  0 
1  1 
1  2 
1  3 
1  4 
1  5 

run 


Sp  TMSEP 
Cp  TMSEP 
MSE  TMSEP 


TMSEPs  FOR 


2 . 51 92862E-03 
6. 2425360E-03 
6. 7220321 E-03 
8. 6688502E-03 
1 . 0253071 E-02 
1 . 1 382802E-02 
1 . 3577277E-02 
1 . 5074901 E-02 
1 . 5540805E-02 
1 . 5885668E-02 
1 . 7365742E-02 
2 . 461 6949E-02 
2. 6280256E-02 
2 . 8023722E-02 
2.961 5028E-02 
1  7 


3 . 4056220E-02 
3 . 4056220E-02 
3. 4056220E-02 


RUN  1  7 

2. 321 4938E-03 
2.5332391 E-03 
2.61 55079E-03 
3 . 1 58861 8E-03 
7 . 8641 446E-03 
1 .2681 485E-02 
1 . 5965864E-02 
2. 8908700E-02 
3 . 02767 1 6E-02 
3 . 27741 88E-02 
3. 3868384E-02 
3. 4056220E-02 
3 .9103031 E-02 
4.348931 1 E-02 
4. 7426648E-02 
1  8 


1 . 1 656721 E-02 
1 . 1 656721 E-02 
1 . 1656721 E-02 


RUN  1 8 


98 


I 


1 

8.814491 3E-03 

2 

9. 2402538E-03 

3 

9. 3459627E-03 

4 

1 .04301 53E-02 

5 

1 . 0963820E-02 

6 

1 . 1 204778E-02 

7 

1 . 1 656721 E-02 

8 

1.351 2224E-02 

9 

1 . 3699356E-02 

1  0 

1 . 51 92608E-02 

1  1 

1 . 5793409E-02 

1  2 

1 . 6079456E-02 

1  3 

1 .7682591 E-02 

1  4 

1 . 8760057E-02 

1  5 

2.511 0584E-02 

run 

1  9 

Sp 

TMSEP  = 

2. 85251 21 E-03 

Cp 

TMSEP 

2 . 85251 21 E-03 

MSE 

TMSEP  = 

8. 1 01 4968E-03 

TMSEPs  FOR  i 

RUN  1 9 

1 

2. 85251  21  E-03 

2 

4. 4923639E-03 

3 

4 . 9251  076E-03 

4 

4. 9598929E-03 

5 

8. 1  01 4968E-03 

6 

8.1  1  87468E-03 

7 

8. 4805330E-03 

8 

8.7108472E-03 

9 

8.9461999E-03 

1  0 

1 . 031  2763E-02 

1  1 

1.211 9596E-02 

1  2 

1  .  31  27306E-02 

1  3 

1  .  3978862E-02 

1  4 

1 . 6505282E-02 

1  5 

2.0519499E-02 

run 

20 

Sp 

TMSEP  = 

3.8725424E-03 

Cp 

TMSEP  = 

3.8725424E-03 

MSE 

TMSEP  = 

2. 373231 8E-02 

TMSEPs  FOR  RUN 


20 


99 


1 

2 . 5775689E-03 

2 

2, 8687967E-03 

3 

3.0975479E-03 

4 

3. 8725424E-03 

5 

5. 0329203E-03 

6 

7 . 2322008E-03 

7 

8.9591 537E-03 

8 

9.231 3904E-03 

9 

1 . 321 7278E-02 

1  0 

1 . 681 7946E-02 

1  1 

1 . 7074332E-02 

1  2 

1 . 9801 777E-02 

1  3 

2 . 1 266723E-02 

1  4 

2.3732318E-02 

1  5 

2. 6426652E-02 

run 

21 

Sp 

TMSEP  = 

1  .  70521  1  4E-02 

Cp 

TMSEP  = 

1  .70521  1  4E-02 

MSE 

TMSEP 

1  .  70521  1  4E-02 

TMSEPs  FOR  RUN 


21 


1 

1  .  70521  1  4E-02 

2 

2. 21 08998E-02 

3 

2. 2773303E-02 

4 

2 . 3853436E-02 

5 

2. 3881 707E-02 

6 

2. 50291 71 E-02 

7 

2.5230104E-02 

8 

2.6611 436E-02 

9 

2. 6725085E-02 

1  0 

2. 7688080E-02 

1  1 

2.7753545E-02 

1  2 

2. 9282084E-02 

1  3 

2.9369567E-02 

1  4 

2.971 5974E-02 

1  5 

3. 1 632055E-02 

run 

22 

Sp 

TMSEP  = 

5. 51  81 938E-03 

Cp 

TMSEP  = 

5.5181938E-03 

MSE 

TMSEP 

5. 51 81 938E-03 

TMSEPs  FOR  RUN 


22 


1  00 


run 


1  3 . 6380207E-03 

2  3.931 8283E-03 

3  3.9480580E-03 

4  4 . 1 61 4547E-03 

5  4. 21 29597E-03 

6  4. 3571 047E-03 

7  5.5181938E-03 

8  5. 71 62573E-03 

9  5. 8647827E-03 

10  6.0956995E-03 

11  6. 8830424E-03 

12  7 . 01 84362E-03 

13  7 . 41 83247E-03 

14  7 . 5743883E-03 

15  1 . 94681 21 E-02 

23 


Sp  TMSEP 
Cp  TMSEP 
MSE  TMSEP 


1 . 2474651 E-02 
1 . 2474651 E-02 
1 . 65861 90E-02 


TMSEPS  FOR  RUN 


23 


run 


1  1  .71  54425E-03 

2  1 . 843361 OE-03 

3  2. 1 247640E-03 

4  3.7305583E-03 

5  7. 36891 25E-03 

6  7 . 64021 98E-03 

7  1 . 0252496E-02 

8  1 . 0763265E-02 

9  1 . 1 81 9728E-02 

10  1 . 2474651 E-02 

11  1 . 3044675E-02 

12  1 .5219216E-02 

13  1 . 65861 90E-02 

14  1 . 7523689E-02 

15  1 .8898807E-02 

24 


Sp 

TMSEP  = 

2.0092988E-02 

Cp 

TMSEP  * 

2.0092988E-02 

MSE 

TMSEP  = 

2.0092988E-02 

TMSEPs  FOR  RUN 


24 


101 


1 

3. 4344932E-03 

2 

6. 0286978E-03 

3 

6. 0683978E-03 

4 

6.0691 400E-03 

5 

6. 21 91 375E-03 

6 

7 . 231 7598E-03 

7 

8. 2883285E-03 

8 

1 . 1 21 6477E-02 

9 

1 . 4045537E-02 

1  0 

1 . 491 551 8E-02 

1  1 

1 . 5253766E-02 

1  2 

1 . 541 7388E-02 

1  3 

1 . 9777540E-02 

1  4 

2 . 0092988E-02 

1  5 

2 . 1 272380E-02 

run 

25 

Sp 

TMSEP 

2.0874506E-03 

Cp 

TMSEP  = 

2 . 0874506E-03 

MSE 

TMSEP  = 

2.000951 2E-02 

TMSEPs  FOR  RUN  25 

1 

2.0874506E-03 

2 

3 . 2037264E-03 

3 

3.9723651E-03 

4 

4. 2093988E-03 

5 

5. 0320607E-03 

6 

5.21 1 9563E-03 

7 

8. 8053672E-03 

8 

1 . 0451 779E-02 

9 

1 . 283791 8E-02 

1  0 

1  .  3056729E-02 

1  1 

1 . 46301 86E-02 

1  2 

2.0009512E-02 

1  3 

2. 1082407E-02 

1  4 

2. 1697044E-02 

1  5 

2.2916408E-02 

run 

26 

Sp 

TMSEP  = 

9.8276651E-04 

Cp 

TMSEP  = 

9.8276651 E-04 

MSE 

TMSEP  = 

9. 8276651 E-04 

TMSEPs  FOR  RUN 


26 


1 

3.661 5500E-04 

2 

4.0323741 E-04 

3 

9.8276651 E-04 

4 

1 . 0234589E-03 

5 

1 . 0259064E-03 

6 

1 . 0351  705E-03 

7 

1  .  1  282420E-03 

8 

1 . 2075821  E-03 

9 

1  .  21  96909E-03 

1  0 

1  .  301  0738E-03 

1  1 

1  .  5202522E-03 

1  2 

1  .  5768777E-03 

1  3 

1  .  7284348E-03 

1  4 

1 . 96851  1 8E-03 

1  5 

1 .20831  32E-02 

run 

27 

Sp 

TMSEP 

=  6.8846606E-03 

Cp 

TMSEP 

=  6.8846606E-03 

MSE 

TMSEP 

=  1 . 9304380E-02 

TMSEPs  FOR 

RUN  27 

1 

6.8846606E-03 

2 

7 . 2969943E-03 

3 

9. 8057995E-03 

4 

1  .  0382390E-02 

5 

1  . 2498997E-02 

6 

1  .  31  36022E-02 

7 

1  .  3230429E-02 

8 

1 . 3853890E-02 

9 

1  .  3952672E-02 

10 

1 .4731946E-02 

1  1 

1  . 70431  75E-02 

1  2 

1 .9304380E-02 

1  3 

2. 0270826E-02 

1  4 

2.0551469E-02 

1  5 

2. 1 667840E-02 

run 

28 

Sp 

TMSEP 

=  3.7091917E-03 

Cp 

TMSEP 

=  3. 7091 91 7E-03 

MSE 

TMSEP 

=  3. 7091 91 7E-03 

TMSEPs  FOR  RUN 


28 


1  03 


1 

3. 5498959E-03 

2 

3 . 7091 91 7E-03 

3 

3 . 81 02050E-03 

4 

3.9531672E-03 

5 

4.0174657E-03 

6 

4.0629935E-03 

7 

4. 0745367E-03 

8 

4 . 0909247E-03 

9 

4 . 2000455E-03 

1  0 

4.2432775E-03 

1  1 

4. 3620034E-03 

1  2 

4. 41 59661 E-03 

1  3 

4. 4462909E-03 

1  4 

4 . 521 841 6E-03 

1  5 

1 . 1 852265E-02 

run 

29 

Sp 

TMSEP  = 

8. 8697784E-03 

Cp 

TMSEP  = 

8, 8697784E-03 

MSE 

TMSEP  = 

8.86977a4E-03 

TMSEPs  FOR  1 

RUN  29 

1 

8.1481 282E-03 

2 

8. 291 9262E-03 

3 

8.8697784E-03 

4 

8.9680441 E-03 

5 

9. 1  282884E-03 

6 

9. 2851 929E-03 

7 

9.31 37734E-03 

8 

9. 5947059E-03 

9 

9. 8677045E-03 

1  0 

9. 91 92383E-03 

1  1 

1 . 0009223E-02 

1  2 

1 . 0221 91 5E-02 

1  3 

1 . 0479798E-02 

1  4 

1 . 0976000E-02 

1  5 

1 . 7647961 E-02 

run 

30 

Sp 

TMSEP  * 

3. 5457979E-03 

cp 

TMSEP  = 

3. 5457979E-03 

MSE 

TMSEP  = 

1  .  3879534E-02 

TMSEPs  FOR  RUN 


30 


1  04 


1 

3.5457979E-03 

2 

4.6322457E-03 

3 

4.8427694E-03 

4 

5.4951 324E-03 

5 

6. 3336785E-03 

6 

6. 3531 036E-03 

7 

7 . 4292999E-03 

8 

8.7361 382E-03 

9 

9.9864267E-03 

1  0 

1 . 1 301 067E-02 

1  1 

1 . 3879534E-02 

1  2 

1 . 4671 805E-02 

1  3 

1 . 4762038E-02 

1  4 

1 . 5681 023E-02 

1  5 

2. 0051 859E-02 

run 

31 

Sp 

TMSEP  = 

1 . 9329343E-02 

Cp 

TMSEP  = 

1 . 9329343E-02 

MSE 

TMSEP  = 

2. 2040404E-02 

TMSEPs  FOR  1 

RUN  31 

1 

1  .  301  2470E-03 

2 

1  .  332801 OE-03 

3 

1 .6352949E-03 

4 

1 .7551 928E-03 

5 

1 . 851  8085E-03 

6 

2.0177322E-03 

7 

4. 581 9799E-03 

8 

4.9842852E-03 

9 

7.8484677E-03 

1  0 

1  .  1  7721 09E-02 

1  1 

1  .  24431 45E-02 

12 

1 .9329343E-02 

1  3 

2.1  689299E-02 

1  4 

2.2040404E-02 

1  5 

2.4905885E-02 

run 

32 

Sp 

TMSEP  = 

5.0999749E-02 

Cp 

TMSEP  = 

5. 6429233E-02 

MSE 

TMSEP  - 

6.2712573E-02 

TNSEPs  FOR  RUN 


32 


1  7 . 7922344E-03 

2  9. 561 7203E-03 

3  9. 6836761 E-03 

4  1 . 1 141410E-02 

5  1 . 41 67450E-02 

6  1 . 4421 574E-02 

7  1 . 5424208E-02 

8  1 . 6907291 E-02 

9  2. 901 0303E-02 

0  2.9186631E-02 

1  3. 201 9943E-02 

2  5. 0999749E-02 

3  5. 51 62951 E-02 

4  5.6429233E-02 

5  6. 271 2573E-02 


run 

33 

• 

Sp 

TMSEP 

8.431 3720E-02 

Cp 

TMSEP  = 

8.431 3720E-02 

MSE 

TMSEP 

8. 431  3720E-02 

TMSEPS  FOR  RUN 


1  4  8 
15  9 


6472956E-02 
9875426E-02 
3888426E-02 
491 3736E-02 
6807437E-02 
0863002E-02 
64061 1 8E-02 
1 674461 E-02 
9072534E-02 
0164599E-02 
61 48276E-02 
2232231 E-02 
631 6334E-02 
431 3720E-02 
0484820E-02 


run 

34 

Sp 

TMSEP  * 

1 .9720267E-03 

Cp 

TMSEP  = 

1 . 9720267E-03 

MSE 

TMSEP  = 

1 .9720267E-03 

TMSEPs  FOR  RUN 


106 


1 

1  .  4564074E-03 

2 

1  -  5348943E-03 

3 

1  .  5788405E-03 

4 

1  .  7205393E-03 

5 

1 . 9720267E-03 

6 

3.0269641 E-03 

7 

3. 1  25601  4E-03 

8 

3. 4055866E-03 

9 

3.6278572E-03 

1  0 

3.9648265E-03 

1  1 

4. 21 02486E-03 

1  2 

5. 5269981  E-03 

1  3 

6. 0082953E-03 

1  4 

6. 6690734E-03 

1  5 

1  .  7902035E-02 

run 

35 

Sp 

TMSEP 

=  2. 21 26485E-02 

Cp 

TMSEP 

=  2.2126485E-02 

MSE 

TMSEP 

=  2.8070798E-02 

TMSEPs  FOR 

RUN  35 

1 

2.1  942048E-03 

2 

3.8941863E-03 

3 

5. 9093987E-03 

4 

5.9594219E-03 

5 

1 . 0993944E-02 

6 

1  .  1  843625E-02 

7 

1  . 2036902E-02 

8 

1  .  324081 4E-02 

9 

1  .  35241  23E-02 

10 

1 . 6991 993E-02 

1  1 

1 . 8741  287E-02 

1  2 

1 .91  71  823E-02 

1  3 

2.2126485E-02 

1  4 

2 . 3006901 E-02 

1  5 

2.8070798E-02 

run 

36 

Sp 

TMSEP 

=  1  . 5084335E-02 

Cp 

TMSEP 

=  1 . 5084335E-02 

MSE 

TMSEP 

=  2. 3686530E-02 

TMSEPs  FOR  RUN 


36 


• 

1 

1  .  5084335E-02 

2 

1 . 5954880E-02 

3 

1  .  727441  9E-02 

4 

1 .7573619E-02 

5 

1 .8410455E-02 

• 

6 

1 .8481053E-02 

7 

1 .93301 1 2E-02 

8 

2.00871 45E-02 

9 

2. 0443222E-02 

1  0 

2. 1  2881  32E-02 

1  1 

2.1 3041 68E-02 

• 

1  2 

2. 3686530E-02 

1  3 

2. 5245925E-02 

1  4 

2. 5533320E-02 

1  5 

2.7481 722E-02 

run 

37 

• 

Sp 

TMSEP 

=  3.8605933E-03 

Cp 

TMSEP 

=  2 . 6786476E-02 

MSE 

TMSEP 

=  2.6786476E-02 

• 

TMSEPs  FOR  1 

RUN  37 

1 

7 . 078851  3E-04 

2 

7 . 31 54463E-04 

3 

1  .  3002879E-03 

4 

2.6231960E-03 

5 

2.96671 36E-03 

• 

6 

2. 9975821 E-03 

7 

3.8605933E-03 

8 

8. 02521 22E-03 

9 

1 .6348951E-02 

1  0 

1 .8301 096E-02 

1  1 

2.0436686E-02 

• 

1  2 

2.28191 19E-02 

1  3 

2.6190016E-02 

1  4 

2. 6786476E-02 

1  5 

3.0853484E-02 

run 

38 

• 

Sp 

TMSEP 

-  1 .0595867E-02 

Cp 

TMSEP 

-  1 .0595867E-02 

MSE 

TMSEP 

=  2.3941863E-02 

• 

TMSEPs  FOR 

RUN  38 

108 


1 

2.9883636E-03 

2 

4 . 1 7641 69E-03 

3 

4 . 8778988E-03 

4 

9.7571025E-03 

5 

9.99731 57E-03 

6 

1  .  0465559E-02 

7 

1  .0595867E-02 

8 

1  .  41 87485E-02 

9 

1 . 7074063E-02 

1  0 

2.05841 34E-02 

1  1 

2 . 3294877E-02 

1  2 

2. 3941 863E-02 

1  3 

2. 551 6720E-02 

1  4 

2 . 5926596E-02 

1  5 

2 . 7840657E-02 

run 

39 

Sp 

TMSEP 

=  1 . 0622767E-02 

Cp 

TMSEP 

=  1 .0622767E-02 

MSE 

TMSEP 

=  1 . 0622767E-02 

TMSEPs  FOR  RUN 


39 


1 

2 

3 

4 

5 

6 

7 

8 
9 

1  0 
1  1 
1  2 
1  3 
1  4 
1  5 


1  .  1 929536E-03 
1  .  3373954E-03 
1  .  670891  1  E-03 
3. 4250987E-03 
3.506341 3E-03 
4.6463534E-03 
4.6803518E-03 
8.6782407E-03 
8. 7229246E-03 
8.8040661 E-03 
1 . 0622767E-02 
1 . 21 461 95E-02 
1 . 2587945E-02 
1 . 3528628E-02 
1 .7254293E-02 


run 

40 

Sp 

TMSEP 

3. 5547487E-02 

Cp 

TMSEP 

s 

3. 5547487E-02 

MSE 

TMSEP 

S 

4. 1834652E-02 

TMSEPs  FOR 

RUN 

40 

109 


1 

7 . 561 9790E-04 

2 

1.57811 20E-03 

3 

2 . 6893581 E-03 

4 

2.8271 538E-03 

5 

1 . 01 57598E-02 

6 

1 . 0704858E-02 

7 

1 . 1 1 761 47E-02 

8 

1 . 2656823E-02 

9 

2. 0222088E-02 

1  0 

2. 0667935E-02 

1  1 

2. 1 879649E-02 

1  2 

2.8531680E-02 

1  3 

3.0591 035E-02 

1  4 

3. 5547487E-02 

1  5 

4. 1 834652E-02 

run 

41 

Sp 

TMSEP 

2.241 3865E-02 

Cp 

TMSEP 

2.241 3865E-02 

MSE 

TMSEP 

2 . 8072061 E-02 

TMSEPs  FOR  RUN  41 

1 

1 . 0048087E-02 

2 

1 .5616417E-02 

3 

1 . 5880233E-02 

4 

1 .6917937E-02 

5 

1  .  749051 9E-02 

6 

1  .  7648792E-02 

7 

1 .8472508E-02 

8 

1 .9556798E-02 

9 

2.0597108E-02 

1  0 

2. 1678284E-02 

1  1 

2. 241 3865E-02 

1  2 

2.4091654E-02 

1  3 

2.7307892E-02 

1  4 

2.8072061 E-02 

1  5 

3.0794047E-02 

run 

42 

Sp 

TMSEP  » 

4.7981236E-03 

Cp 

TMSEP  = 

4. 7981236E-03 

MSE 

TMSEP  - 

4.7981236E-03 

TMSEPs  FOR  RUN 


42 


no 


3 

4 

5 

6 

7 

8 
9 

1  0 
1  1 
1  2 
1  3 
1  4 
1  5 


run 


Sp  TMSEP  = 
Cp  TMSEP  = 
MSE  TMSEP 


4364881 E-03 
52591 90E-03 
6066370E-03 
91 991 76E-03 
,  06801 61 E-03 
1  261 21 8E-03 
7981 236E-03 
,  2037672E-03 
,  3428058E-03 
,  681 961  3E-03 
.721641 2E-03 
,  0622832E-03 
,  0960101  E-03 
,  4953892E-03 
.  5678339E-02 
43 


3.9310423E-03 
3.9310423E-03 
3. 931  0423E-03 


TMSEPs  FOR  RUN 


43 


run 


1  1 .8738484E-03 

2  2. 3403340E-03 

3  2. 9570037E-03 

4  3 . 1 850252E-03 

5  3.931 0423E-03 

6  4. 7286372E-03 

7  4.8334445E-03 

8  4.8407987E-03 

9  5. 7 1 94503E-03 

10  6. 7090066E-03 

11  7 .6004523E-03 

12  9. 2586232E-03 

13  9. 2892852E-03 

14  1 . 2049303E-02 

15  1 . 2300548E-02 

44 


Sp  TMSEP 
Cp  TMSEP 
MSE  TMSEP 


TMSEPs  FOR  RUN 


1 . 201 91 08E-03 
1.81 401 1 9E-02 
1.81401 1 9E-02 


44 


1  1  1 


1 

3.9169341 E-04 

2 

5. 2826642E-04 

3 

1  .  20-’  91  08E-03 

4 

1 . 2748841 E-03 

5 

3.41 31 568E-03 

6 

6. 1 522294E-03 

7 

6.971 8626E-03 

8 

7 .0441 552E-03 

9 

7 . 1 675945E-03 

1  0 

7 . 9703350E-03 

1  1 

8.1011681 E-03 

1  2 

8.61 36796E-03 

1  3 

1.81401 1 9E-02 

1  4 

1 . 9227074E-02 

1  5 

2.1237271 E-02 

run 

45 

Sp 

TMSEP 

3.0700916E-03 

Cp 

TMSEP  = 

3 . 070091 6E-03 

MSE 

TMSEP 

1  .  1  097923E-02 

TMSEPs  FOR 

RUN  4  5 

1 

3.0700916E-03 

2 

3. 3025786E-03 

3 

4.1 044764E-03 

4 

5. 5306656E-03 

5 

5. 6966278E-03 

6 

5.9962580E-03 

7 

6.9032218E-03 

8 

7 . 6960446E-03 

9 

1 . 0958084E-02 

1  0 

1 . 1 097923E-02 

1  1 

1 . 1 505049E-02 

1  2 

1  .  1 665934E-02 

1  3 

1  .  1997189E-02 

1  4 

1 . 2202757E-02 

1  5 

1 . 3101 306E-02 

run 

46 

sp 

TMSEP  - 

2.2392911 E-03 

cp 

TMSEP  = 

8.1 968531 E-03 

MSE 

TMSEP  = 

1  .  1  323700E-02 

TMSEPs  FOR  RUN 


46 


1 

2. 239291 1 E-03 

2 

2 . 4520580E-03 

3 

2.9001222E-03 

4 

5. 24661 60E-03 

5 

5. 2603823E-03 

6 

5. 31 521 07E-03 

7 

6. 3047521 E-03 

8 

6.606371 1 E-03 

9 

7.9144575E-03 

1  0 

8. 1968531 E-03 

1  1 

8. 2538687E-03 

1  2 

9. 0868426E-03 

1  3 

1 . 1 323700E-02 

1  4 

1 . 3579807E-02 

1  5 

1 . 5681 304E-02 

run 

47 

Sp 

TMSEP  = 

1 

. 0096396E-02 

Cp 

TMSEP 

1 

.0096396E-02 

MSE 

TMSEP 

1 

. 0096396E-02 

TMSEPs  FOR  1 

RUN  47 

1 

9. 9929059E-03 

2 

1 . 0018086E-02 

3 

1  .  0096396E-02 

4 

1  .  0306062E-02 

5 

1  .  1  321  264E-02 

6 

1  .  1  852885E-02 

7 

1  -  2029038E-02 

8 

1  .  21 281  58E-02 

9 

1  . 21  28995E-02 

10 

1  .  2503473E-02 

1  1 

1  .  275731  8E-02 

1  2 

1  .  421  7354E-02 

1  3 

1  .  4767253E-02 

1  4 

1  .627  1  524E-02 

1  5 

2. 87891  50E-02 

run 

48 

Sp 

TMSEP  » 

7 . 3499735E-03 

Cp 

TMSEP  = 

7 . 3499735E-03 

MSE 

TMSEP  = 

1 . 943331 4E-02 

TMSEPs  FOR  RUN 


48 


1  1  3 


• 

1 

7 . 3499735E-03 

2 

7 . 841  94'75E-03 

3 

1 . 0533275E-02 

4 

1  .  1  1  74344E-02 

5 

1.11 79664E-02 

• 

6 

1 . 1 932554E-02 

7 

1 . 51 24455E-02 

8 

1  .  5476356E-02 

9 

1 . 5993901 E-02 

10 

1 . 6502503E-02 

1  1 

1  .  7872537E-02 

• 

1  2 

1  .  943331  4E-02 

1  3 

2 . 0548729E-02 

1  4 

2.0941 224E-02 

1  5 

2 . 22051  24E-02 

run 

49 

• 

Sp 

TMSEP 

=  4. 21 70435E-02 

Cp 

TMSEP 

=  4 . 21  70435E-02 

MSE 

TMSEP 

=  4 . 21 70435E-02 

TMSEPs  FOR  RUN 


49 


1 

1 . 5701 497E-02 

2 

1 . 6772849E-02 

3 

1 .7934496E-02 

4 

1 .9688854E-02 

5 

2 . 1 170381 E-02 

6 

2.2765471 E-02 

7 

2 . 4873454E-02 

8 

2. 581 9845E-02 

9 

2.7326189E-02 

10 

2. 91 26355E-02 

1  1 

3.081 3400E-02 

1  2 

3. 1698938E-02 

1  3 

3. 3878606E-02 

1  4 

4.2170435E-02 

1  5 

4.491 1824E-02 

run 

50 

Sp  TMSEP 
Cp  TMSEP 
MSE  TMSEP 


2.2768520E-02 
2. 2768520E-02 
3.6368068E-02 


TMSEPs  FOR  RUN 


50 


1  1  4 


1 

1 . 5540421 E-02 

2 

1 . 6995009E-02 

3 

1 .7393902E-02 

4 

2.0387221 E-02 

5 

2 . 1 287283E-02 

6 

2. 1 346433E-02 

7 

2 . 21 53363E-02 

8 

2. 2768520E-02 

9 

2 . 3359068E-02 

1  0 

2 . 4774464E-02 

1  1 

2.57701 80E-02 

1  2 

2. 891 901 9E-02 

1  3 

3 . 0525209E-02 

1  4 

3 . 6368068E-02 

1  5 

3.8798217E-02 

run 

51 

Sp 

TMSEP 

2. 4604530E-03 

Cp 

TMSEP  = 

2 . 4604530E-03 

MSE 

TMSEP 

7.01 50704E-03 

TMSEPs  FOR 

RUN  51 

1 

1 .6393900E-03 

2 

1  .  8884840E-03 

3 

2 . 26061  49E-03 

4 

2 . 4604530E-03 

5 

2 . 4627568E-03 

6 

2 . 6644068E-03 

7 

2.7719922E-03 

8 

3 . 1 81 8897E-03 

9 

3. 1 976223E-03 

1  0 

3 . 4056555E-03 

1  1 

7 . 01 50704E-03 

1  2 

7. 1953204E-03 

1  3 

7.7710031E-03 

1  4 

7 . 9887751 E-03 

15 

2.0532234E-02 

run 

52 

Sp 

TMSEP  = 

1 . 5042799E-02 

Cp 

TMSEP  = 

3. 2231 480E-02 

MSE 

TMSEP  = 

3.2231480E-02 

TMSEPs  FOR  RUN 


52 


1  1  5 


• 

1 

1.351031 3E-02 

2 

1  .  4268238E-02 

3 

1 . 5042799E-02 

4 

1.531 5600E-02 

5 

1  .  6465578E-02 

• 

6 

1  .  7620863E-02 

7 

1 .8399971 E-02 

8 

1 . 875601 9E-02 

9 

2.1 3301 54E-02 

1  0 

2. 406201 7E-02 

1  1 

2.459041 OE-02 

• 

1  2 

2.6639560E-02 

1  3 

2.7676962E-02 

1  4 

3 . 2231  480E-02 

1  5 

3 . 5307933E-02 

run 

53 

• 

Sp  TMSEP 

=  4 . 1  374839E-03 

Cp  TMSEP 

=  4. 1 374839E-03 

MSE  TMSEP 

=  4 . 1  374839E-03 

• 

TMSEPs  FOR 

RUN  5  3 

1 

1  .  41  67477E-03 

2 

1 . 8064859E-03 

3 

1  .  81 2721 5E-03 

4 

4. 1 374839E-03 

5 

5. 366351 5E-03 

# 

6 

5. 3790277E-03 

7 

5. 4524424E-03 

8 

5.7910713E-03 

9 

6. 2028468E-03 

1  0 

6.9525018E-03 

1  1 

8.6443033E-03 

# 

1  2 

8. 733331 2E-03 

1  3 

9.9164629E-03 

1  4 

1 .0268040E-02 

1  5 

1 . 1 2673^7E-02 

run 

54 

• 

Sp 

TMSEP 

=  1 . 4347644E-02 

Cp 

TMSEP 

=  1  .  4347644E-02 

MSE 

TMSEP 

«  1 . 4347644E-02 

• 

TMSEPs  FOR 

RUN  54 

116 


1 

5.841 5225E-03 

2 

6. 4000203E-03 

3 

7 . 0034480E-03 

4 

8. 8544022E-03 

5 

9. 1 899643E-03 

6 

9. 4587682E-03 

7 

9- 9534485E-03 

8 

1 . 0385258E-02 

9 

1 . 085973CE-02 

1  0 

1 . 1 279927E-02 

1  1 

1 . 41 4621 2E-02 

1  2 

1 . 4347644E-02 

1  3 

1 . 53661 05E-02 

1  4 

1 . 761 6469E-02 

1  5 

1 . 8684378E-02 

run 

55 

Sp 

TMSEP  = 

1  .  2993388E-02 

Cp 

TMSEP  = 

1  . 2993388E-02 

MSE 

TMSEP  = 

1 . 91 65561 E-02 

TMSEPs  FOR  RUN  55 

1 

4.9696909E-03 

2 

7.7763824E-03 

3 

9.6101 258E-03 

4 

1  .  1 6967 1  5E-02 

5 

1  .  21 70565E-02 

6 

1.27161 34E-02 

7 

1  .  2899847E-02 

8 

1  .  2993388E-02 

9 

1 . 501  5327E-02 

1  0 

1  .  555541  7E-02 

1  1 

1 . 5641  246E-02 

1  2 

1 .6055508E-02 

1  3 

1  .  78381 84E-02 

1  4 

1 . 9165561 E-02 

1  5 

2 . 1  346293E-02 

run 

56 

Sp 

TMSEP  = 

6. 3008070E-02 

Cp 

TMSEP  = 

6. 3008070E-02 

MSE 

TMSEP  = 

7. 3209882E-02 

TMSEPs  FOR  RUN 


56 


1 

6.1  677363E-02 

2 

6. 2864579E-02 

3 

6. 3008070E-02 

4 

6.6310331 E-02 

5 

6. 6765495E-02 

6 

6.68531 1 3E-02 

7 

7.  1422204E-02 

8 

7 . 1  454287E-02 

9 

7. 3190719E-02 

1  0 

7 . 3209882E-02 

1  1 

7 . 5800024E-02 

1  2 

7 . 7444568E-02 

1  3 

7 . 8586839E-02 

1  4 

7 . 9587929E-02 

1  5 

8. 5475355E-02 

run 

57 

Sp 

TMSEP 

=  3.7371 186E-03 

Cp 

TMSEP 

=  3.7371 186E-03 

MSE 

TMSEP 

=  3.7371 186E-03 

TMSEPs  FOR 

RUN  57 

1 

3.7371 186E-03 

2 

3.8739347E-03 

3 

3.9046754E-03 

4 

4.03831 20E-03 

5 

4. 1861 599E-03 

6 

4.4727097E-03 

7 

4.6059825E-03 

8 

4.7026719E-03 

9 

5.1 347492E-03 

1  0 

5. 8260681 E-03 

1  1 

5.8471588E-03 

1  2 

6. 2553254E-03 

1  3 

6. 449021 4E-03 

1  4 

7.0934095E-03 

1  5 

9. 5695062E-03 

run 

58 

Sp 

TMSEP 

==  1 . 2361  378E-02 

Cp 

TMSEP 

*  1  .  2361  378E-02 

MSE 

TMSEP 

=  1 .2361  378E-02 

TMSEPs  FOR  RUN 


58 
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1 

2. 4738992E-03 

2 

3.7456378E-03 

3 

3.8988995E-03 

4 

4. 20701 22E-03 

5 

4. 5560552E-03 

6 

4 . 8246444E-03 

7 

6. 3058720E-03 

8 

1  .  2361  378E-02 

9 

1  .  3068646E-02 

1  0 

1  .  3421  705E-02 

1  1 

1  .  3535345E-02 

1  2 

1 . 41 64726E-02 

1  3 

1  .  6804852E-02 

1  4 

1  .  7755972E-02 

1  5 

2. 3304023E-02 

run 

59 

Sp 

TMSEP 

=  1 . 51 1 0006E-02 

Cp 

TMSEP 

=  1 , 51 1 0006E-02 

MSE 

TMSEP 

=  1  .  51  1 0006E-02 

TMSEPs  FOR 

RUN  59 

1 

1 . 01 061  59E-02 

2 

1.11 30731 E-02 

3 

1  .  1  358641  E-02 

4 

1  .  1  672366E-02 

5 

1  .  247  1 750E-02 

6 

1  .  3456321  E-02 

7 

1 . 3687690E-02 

8 

1.511 0006E-02 

9 

1  .  5243942E-02 

1  0 

1.71 361 47E-02 

1  1 

1  .  8399373E-02 

1  2 

1 . 8740477E-02 

1  3 

1 . 974821 8E-02 

1  4 

2.0658856E-02 

1  5 

2. 3248075E-02 

run 

60 

Sp  TMSEP 
Cp  TMSEP 
MSE  TMSEP 


1 .8573448E-02 
1 . 8573448E-02 
1 .8573448E-02 


TMSEPs 


I 


FOR  RUN  60 

1  1 . 1 594909E-02 

2  1 . 2438751 E-02 

3  1 . 2594829E-02 

4  1 . 2842594E-02 

5  1 . 3457467E-02 

6  1 . 3940695E-02 

7  1 . 5752792E-02 

8  1 . 8573448E-02 

9  2. 0086005E-02 

10  2. 241 4638E-02 

11  2. 3428341 E-02 

12  2. 4330338E-02 

13  2. 5231 1 92E-02 

14  2. 5254766E-02 

15  2. 7355636E-02 
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